feat(config-upgrade-snapshot): live config-upgrade soak infra + fix bug-1468/bug-1469 (task 8.6)
Build the live 3-node chaotic-multi-kill config-upgrade soak scenario (run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1, 6.4. This needed genuinely new client-side infrastructure: a hand-rolled capnp send path for OP_upgradeConfig (the auto-generated encoder is a no-op for its Data fields, worked around the same way xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig CLI commands. Building and running this live surfaced two real production bugs, both fixed here: - bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's primary/standby dispatch guard in cluster.c, making config upgrades unreachable from any network client. - bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called config_activate_committed(), so the node that originates a config upgrade never refreshed its own in-memory state as part of committing it. Killing that node before some later, unrelated event happened to refresh it left its checkpoint permanently stale -- the first live, empirical reproduction of bug-857's originally hypothesized failure mode. Fixed by activating before the marker-triggered checkpoint write, since checkpoint content is built from live runtime tables that only reflect the reload after activation runs. A separate, narrower checkpoint divergence (bug-1471, not config-related) remains open and still blocks a clean soak PASS -- task 8.6 stays in-progress until it's resolved.
This commit is contained in:
parent
2bd44e8e4d
commit
959b0cdacb
13 changed files with 1022 additions and 10 deletions
|
|
@ -266,6 +266,83 @@ invariant is asserted with an immediate `fail()` inside
|
|||
`run_array_chaos_retry_round()`, so a violation aborts the run before this
|
||||
line is ever reached.
|
||||
|
||||
### `--config-upgrade` mode: live configuration upgrade under chaotic failover (config-upgrade-snapshot task 8.6)
|
||||
|
||||
Drives a real `OP_upgradeConfig` RPC (the new `upgradeconfig` `xce_test`
|
||||
client command, backed by `xcecli_upgradeconfig` →
|
||||
`xce_upgradeconfig_req_raw`) against the live primary, submitted in the
|
||||
**background** and interleaved with a bounded chaotic kill/restart sequence
|
||||
(`XCE_HA_CONFIGUPGRADE_KILLS` rounds, default 3) — relying on the client's own
|
||||
transparent failover (`XCECLI_FAILOVER_CALL`) to survive a primary change
|
||||
mid-call, exactly like `--array-chaos-retry` relies on it for a job-array
|
||||
submit. See `run_config_upgrade_round()`'s header comment (`run-ha-soak.sh`)
|
||||
for the full mechanism trace.
|
||||
|
||||
Sequence:
|
||||
|
||||
1. Joins a synthetic worker host (wire-level `join`, same pattern
|
||||
`--factstream` uses — no real `xcecd` process) and admin-closes it via the
|
||||
new `hostctrl -c` command, giving Req 6.4 a concrete admin bit to prove
|
||||
survives the upgrade.
|
||||
2. Submits an initial batch of jobs — committed state the upgrade must
|
||||
preserve.
|
||||
3. Builds a second, "upgrade-target" `cluster.bin` from a scratch copy of the
|
||||
**whole** `src/configs/` tree (not just `queues.yaml` — `xceadmin config
|
||||
build` needs the full directory) with one deliberate, observable change:
|
||||
the `night` queue's description text. Deliberately an **update to an
|
||||
existing queue**, not an added one — see `run_config_upgrade_round()`'s
|
||||
own comment and `.wolf/buglog.json` bug-1469: a live (non-restart) config
|
||||
upgrade has no working mechanism today to create a *new* runtime queue
|
||||
object in a member's `c->qmgr` (`build_queues()`, queue.c, only ever runs
|
||||
once at process boot); that is a genuine, deep gap in tasks 1-7's
|
||||
materialization logic, confirmed live and flagged for their owners, but
|
||||
out of this task's scope to fix. An update to an existing queue's
|
||||
description exercises exactly what Req 4.1-4.3/6.1/6.4 ask this scenario
|
||||
to prove without depending on that separate, unrelated mechanism.
|
||||
4. Submits the upgrade in the background, then runs the chaotic kill/restart
|
||||
sequence — restarting the previously-downed node WITHOUT waiting for full
|
||||
reconvergence before the next kill (like `--chaotic`), so the upgrade
|
||||
genuinely has a chance to interleave with an in-progress recovery, not just
|
||||
complete cleanly before or after every kill.
|
||||
5. After the cluster reconverges 3-of-3 and the background submission
|
||||
confirms success (retried once against the settled cluster if it did not
|
||||
report success while kills were still landing), hard-asserts (`fail()`,
|
||||
not count-and-tolerate — a definite correctness claim, like
|
||||
`--force-snapshot`):
|
||||
- **materialized-state identity** across all 3 members — byte-diff
|
||||
(`snapdiff`) of each member's latest on-disk checkpoint file;
|
||||
- **active-configuration identity** — `configcompare` (task 8.4's tool,
|
||||
its first live use here) across all 3 members' `conf/current`;
|
||||
- the synthetic host's **admin-CLOSED status specifically survives**,
|
||||
checked DIRECTLY per member (not merely inferred from the two identity
|
||||
checks above) by rotating primary through all 3 nodes in turn
|
||||
(`force_elect_node`) and asserting `workerlist` shows it CLOSED while
|
||||
each one serves — proving a standby-turned-primary materializes the
|
||||
admin bit correctly too, not just whichever node happened to be primary
|
||||
when the upgrade committed.
|
||||
|
||||
Single dedicated round, not a loop — like `--force-snapshot`/`--array-soak`.
|
||||
|
||||
```bash
|
||||
SKILL_DIR=.claude/skills/xce-test-ha
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --config-upgrade ~/tmp/xce-ha-configupgrade
|
||||
# tune kill-round count / synthetic host name
|
||||
XCE_HA_CONFIGUPGRADE_KILLS=5 XCE_HA_CONFIGUPGRADE_HOST=my-host \
|
||||
nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --config-upgrade ~/tmp/xce-ha-configupgrade
|
||||
```
|
||||
|
||||
Verdict line:
|
||||
|
||||
```
|
||||
[xce-test-ha] PASS (config-upgrade): 3 jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members
|
||||
```
|
||||
|
||||
Like `--array-soak`/`--array-chaos-retry`, `result` is unconditionally
|
||||
`PASS` here — every hard invariant is asserted with an immediate `fail()`
|
||||
inside `run_config_upgrade_round()`, so a violation aborts the run before
|
||||
this line is ever reached; there is no `verdict_emit`/bug-924 exclusion
|
||||
tally for this mode.
|
||||
|
||||
### Verdict self-test (no cluster)
|
||||
|
||||
The verdict parser has a deterministic, pure-bash self-test that feeds fixture
|
||||
|
|
@ -361,7 +438,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| Env var | Meaning | Default |
|
||||
|---|---|---|
|
||||
| `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, or `array-chaos-retry` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, `array-chaos-retry`, or `config-upgrade` (its `--flag` wins) | `sequential` |
|
||||
| `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `<repo>/src/xce.build/controller/xce_controller` |
|
||||
| `XCE_HA_XT` | `xce_test` binary | `<repo>/src/xce.build/controller/xce_test` |
|
||||
| `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` |
|
||||
|
|
@ -381,6 +458,8 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK
|
|||
| `XCE_HA_ARRAYRETRY_ROUNDS` | `--array-chaos-retry` kill-during-submit rounds | `5` |
|
||||
| `XCE_HA_ARRAYRETRY_SIZE` | `--array-chaos-retry` element count of the array each round submits | `6` |
|
||||
| `XCE_HA_ARRAYRETRY_KILL_DELAY` | `--array-chaos-retry` seconds slept after backgrounding the submit before killing the primary | `0` |
|
||||
| `XCE_HA_CONFIGUPGRADE_KILLS` | `--config-upgrade` kill/restart rounds interleaved with the upgrade submission | `3` |
|
||||
| `XCE_HA_CONFIGUPGRADE_HOST` | `--config-upgrade` name of the synthetic worker host joined and admin-closed before the upgrade | `cfgupgrade-host` |
|
||||
|
||||
## What the driver does (maps to the requested steps)
|
||||
|
||||
|
|
|
|||
|
|
@ -32,6 +32,25 @@
|
|||
# own log that a snapshot install actually happened (not
|
||||
# just a claim), then asserts zero walcompare divergence
|
||||
# and zero job loss. Single dedicated round, not a loop.
|
||||
# --config-upgrade config-upgrade-snapshot task 8.6 (Req 4.1, 4.3, 6.1,
|
||||
# 6.4): admin-closes a synthetic worker host, then submits a
|
||||
# live configuration upgrade (OP_upgradeConfig, via the new
|
||||
# `upgradeconfig` xce_test command) CONCURRENTLY with a
|
||||
# bounded chaotic kill/restart sequence (XCE_HA_CONFIGUPGRADE_
|
||||
# KILLS rounds), relying on the client's own transparent
|
||||
# failover (XCECLI_FAILOVER_CALL) to survive a primary
|
||||
# change mid-call. After the cluster reconverges, hard-
|
||||
# asserts (fail(), not count-and-tolerate — like
|
||||
# --force-snapshot, this is a definite correctness claim):
|
||||
# (1) materialized-state identity across all 3 members via
|
||||
# byte-diff of each member's latest on-disk checkpoint file
|
||||
# (snapdiff); (2) active-configuration identity via
|
||||
# configcompare (task 8.4's tool, its first live use); (3)
|
||||
# the admin-closed host's status specifically survives,
|
||||
# checked DIRECTLY per member (not inferred from (1)/(2))
|
||||
# by rotating primary through all 3 nodes and querying
|
||||
# workerlist against each. Single dedicated round, not a
|
||||
# loop.
|
||||
# --factstream raft-apply-decoupling task 7.2 (Req 10.2/10.4): drives a
|
||||
# continuous WORKER-FACT stream (jstart/jfinish/jfail — real
|
||||
# wire-protocol worker reports, distinct from job-submission
|
||||
|
|
@ -154,6 +173,14 @@
|
|||
# (default 0 — immediate, to maximize "chaotic"
|
||||
# uncertainty about whether the WAL append/
|
||||
# commit happened before the kill)
|
||||
# XCE_HA_CONFIGUPGRADE_KILLS kill/restart rounds interleaved with the
|
||||
# config-upgrade submission (default 3,
|
||||
# independent of the general --chaotic ROUNDS
|
||||
# knob — this is a single dedicated round, not
|
||||
# a soak loop)
|
||||
# XCE_HA_CONFIGUPGRADE_HOST name of the synthetic worker host joined and
|
||||
# admin-closed before the upgrade, used to prove
|
||||
# Req 6.4 (default "cfgupgrade-host")
|
||||
set -u
|
||||
|
||||
# ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─
|
||||
|
|
@ -174,6 +201,7 @@ while [ $# -gt 0 ]; do
|
|||
--factstream) MODE="factstream" ;;
|
||||
--array-soak) MODE="array-soak" ;;
|
||||
--array-chaos-retry) MODE="array-chaos-retry" ;;
|
||||
--config-upgrade) MODE="config-upgrade" ;;
|
||||
--selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;;
|
||||
*[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir
|
||||
*) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds
|
||||
|
|
@ -1016,6 +1044,276 @@ run_force_snapshot_round() {
|
|||
walcheck_count "${#ALL_JOBIDS[@]}"
|
||||
}
|
||||
|
||||
# ── Config-upgrade primitives (config-upgrade-snapshot task 8.6) ────────────
|
||||
|
||||
# Locate the xceadmin binary — same search list deploy-ha.sh uses (meson
|
||||
# build output under whichever profile built it, or a nix-installed prefix).
|
||||
# Needed here because this mode builds a SECOND cluster.bin itself (the
|
||||
# upgrade target), unlike every other mode which only ever consumes the one
|
||||
# deploy-ha.sh already built into ctrl-1/conf during Step 2.
|
||||
find_xceadmin() {
|
||||
local candidate
|
||||
for candidate in \
|
||||
"$REPO/src/xce.build/xceadmin/.venv/bin/xceadmin" \
|
||||
"$REPO/src/build.release/xceadmin/.venv/bin/xceadmin" \
|
||||
"$REPO/src/build.dbg/xceadmin/.venv/bin/xceadmin" \
|
||||
"$REPO/src/result/bin/xceadmin" \
|
||||
"$(command -v xceadmin 2>/dev/null || true)"
|
||||
do
|
||||
if [ -n "$candidate" ] && [ -x "$candidate" ]; then
|
||||
echo "$candidate"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Echo a member's latest (highest-LSN) on-disk checkpoint file
|
||||
# ($TOP/controller/<node>/data/snap/snap-<lsn hex>.capnp — persist.c's
|
||||
# p->snap_path, "%s/data/snap" under the node's workdir/name), or fail with
|
||||
# rc!=0 if none exists yet. Filenames are fixed-width %016 hex, so
|
||||
# lexicographic sort == LSN-numeric sort.
|
||||
latest_snapshot_file() {
|
||||
local node="$1" f
|
||||
f="$(ls -1 "$TOP/controller/$node/data/snap"/snap-*.capnp 2>/dev/null | sort | tail -1)"
|
||||
[ -n "$f" ] || return 1
|
||||
echo "$f"
|
||||
}
|
||||
|
||||
# config-upgrade-snapshot task 8.6 (Req 4.1, 4.3, 6.1, 6.4): drive a live
|
||||
# configuration upgrade (OP_upgradeConfig, the new client built in this same
|
||||
# task — lib/request.c xce_upgradeconfig_req_raw / lib/xceapi.c
|
||||
# xce_upgradeconfig / lib/client.c xcecli_upgradeconfig, testsuite/test.c
|
||||
# cmd_upgradeconfig) while chaotically killing/restarting the primary, then
|
||||
# hard-assert (fail(), not count-and-tolerate — mirroring
|
||||
# run_force_snapshot_round's own convention: this is a definite correctness
|
||||
# claim, not a racy/sometimes-expected outcome class):
|
||||
#
|
||||
# - materialized-state identity across all 3 members (Req 4.1) — via
|
||||
# byte-diff (snapdiff) of each member's latest on-disk checkpoint file,
|
||||
# the SAME tooling design.md's Requirements Traceability row 6.1 cites
|
||||
# (snapdiff_paths, configcompare) — NOT live per-node joblist/workerlist
|
||||
# RPCs, because this system's read model is primary-only: OPCode_get
|
||||
# (which joblist/workerlist/reslist all use) is absent from
|
||||
# ha_dispatch_gate's CTRL_STANDBY allow-list (ha.c), so a standby cannot
|
||||
# answer it directly — confirmed by direct inspection, not assumed.
|
||||
# - active-configuration identity across all 3 members (Req 4.2/4.3) — via
|
||||
# configcompare (task 8.4's tool), its first live use.
|
||||
# - a host's administratively-set status specifically survives (Req 6.4),
|
||||
# checked DIRECTLY per member — not merely inferred from the aggregate
|
||||
# identity checks above — by rotating primary through each of the 3
|
||||
# nodes in turn (force_elect_node, already used for this script's
|
||||
# leadership-coverage step) and querying workerlist against whichever
|
||||
# node is currently serving, asserting the host shows CLOSED each time.
|
||||
#
|
||||
# One dedicated round, not a loop — like run_force_snapshot_round.
|
||||
run_config_upgrade_round() {
|
||||
local kills="${XCE_HA_CONFIGUPGRADE_KILLS:-3}"
|
||||
local host="${XCE_HA_CONFIGUPGRADE_HOST:-cfgupgrade-host}"
|
||||
local xceadmin scratch_dir scratch_bin
|
||||
|
||||
xceadmin="$(find_xceadmin)" \
|
||||
|| fail "config-upgrade: xceadmin binary not found (run /xce-build all or /xce-build xceadmin)"
|
||||
|
||||
# 1. Join + admin-close a synthetic worker host BEFORE the upgrade, so
|
||||
# Req 6.4 has a concrete admin bit to prove survives. Wire-level
|
||||
# synthetic worker join — the same pattern run_factstream_round
|
||||
# already uses (`set myname <host>` + `worker` + `join`); no real
|
||||
# xcecd process is ever started. do_hostcontrol_op (worker.c) requires
|
||||
# the host to already exist in the live worker table (-ENOENT
|
||||
# otherwise) — this join step exists precisely to satisfy that, since
|
||||
# hosts.yaml config entries do NOT auto-populate workermgr (a worker_t
|
||||
# is created only by a live join, requirements.md's own Boundary
|
||||
# Context note).
|
||||
log "config-upgrade: joining synthetic host $host"
|
||||
local jout
|
||||
jout="$(printf 'set masters %s\nset myname %s\nworker\njoin\nquit\n' "$MASTERS" "$host" | xtest)"
|
||||
echo "$jout" | grep -q "host join ok" \
|
||||
|| fail "config-upgrade: synthetic host $host failed to join: $jout"
|
||||
|
||||
log "config-upgrade: admin-closing $host via hostctrl"
|
||||
local hout
|
||||
hout="$(printf 'set masters %s\nset myname soak\nclient\nwait-primary %s\nhostctrl -c -h %s\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$host" | xtest)"
|
||||
echo "$hout" | grep -q "Operation on host $host done" \
|
||||
|| fail "config-upgrade: hostctrl -c on $host failed: $hout"
|
||||
log "config-upgrade: $host admin-closed"
|
||||
|
||||
# 2. Initial batch of jobs — committed state the upgrade must preserve.
|
||||
submit_jobs "$NJOBS" || fail "config-upgrade: initial submit failed"
|
||||
verify_no_loss || fail "config-upgrade: job loss before the upgrade started"
|
||||
|
||||
# 3. Build the upgrade-target config: a scratch COPY of the whole
|
||||
# src/configs/ tree (schema/ subdir included, exactly mirroring what
|
||||
# deploy-ha.sh itself passes to `xceadmin config build`) with one
|
||||
# deliberate, observable change — the "night" queue's description
|
||||
# text — so the post-upgrade identity check is a real assertion, not
|
||||
# a tautology.
|
||||
#
|
||||
# Deliberately an UPDATE to an EXISTING queue, not an ADDED queue
|
||||
# (bug-1469 finding, .wolf/buglog.json): a live (non-restart) config
|
||||
# upgrade has no working mechanism to create a NEW runtime queue_t
|
||||
# object in any member's c->qmgr — build_queues() (queue.c), the only
|
||||
# code that ever populates c->qmgr from parsed config, runs exactly
|
||||
# once at process boot (xce_config_init) and is unconditional/blind-
|
||||
# create, unsafe to re-invoke live. That is a genuine, deep,
|
||||
# confirmed gap in tasks 1-7's materialization logic (requires new,
|
||||
# carefully-designed production code plus a queue-removal-semantics
|
||||
# design decision this task is not chartered to make) — reproduced
|
||||
# live TWICE (before and after config-upgrade-snapshot task 6.2's own
|
||||
# activation-ordering fix), fully root-caused, and flagged for
|
||||
# tasks 1-7's owners, but explicitly OUT OF SCOPE for task 8.6 to fix.
|
||||
# An UPDATE to an existing queue's description exercises exactly what
|
||||
# Requirements 4.1-4.3/6.1/6.4 ask this task to prove (materialized
|
||||
# job/host state + active-configuration identity across members,
|
||||
# surviving chaotic failover) without depending on that separate,
|
||||
# unrelated, unfixed mechanism — collect_queue_frames' QueueAdminState
|
||||
# frame only ever carries name+admin-status (persist.c:1895), never
|
||||
# description/priority/etc., so this change is invisible to the
|
||||
# queue-entity section either way and is carried purely by the
|
||||
# checkpoint's embedded raw-config-bytes section (task 2.3,
|
||||
# read_active_config_bytes — always a fresh disk re-read, unaffected
|
||||
# by the qmgr-sync gap) and by configcompare's config-file-hash
|
||||
# comparison — both of which are exactly the "active configuration
|
||||
# identical" mechanisms Requirement 4.2/4.3 actually asks about.
|
||||
scratch_dir="$TOP/.cfgupgrade-src"
|
||||
scratch_bin="$TOP/.cfgupgrade-src.bin"
|
||||
rm -rf "$scratch_dir" "$scratch_bin"
|
||||
cp -r "$REPO/src/configs" "$scratch_dir"
|
||||
sed -i 's/description: "night queue"/description: "night queue (config-upgrade-snapshot task 8.6 soak marker)"/' \
|
||||
"$scratch_dir/queues.yaml"
|
||||
grep -q "task 8.6 soak marker" "$scratch_dir/queues.yaml" \
|
||||
|| fail "config-upgrade: sed did not find the expected 'night queue' description line to mark — src/configs/queues.yaml may have changed shape"
|
||||
"$xceadmin" config build "$scratch_dir" -o "$scratch_bin" \
|
||||
|| fail "config-upgrade: xceadmin config build failed for the upgrade-target config"
|
||||
log "config-upgrade: built upgrade-target cluster.bin with an updated 'night' queue description: $scratch_bin"
|
||||
|
||||
# 4. Submit the upgrade CONCURRENTLY with a bounded kill/restart
|
||||
# sequence — at least one kill/restart genuinely overlaps the upgrade
|
||||
# RPC. xcecli_upgradeconfig's XCECLI_FAILOVER_CALL wrapper (Part A) is
|
||||
# relied on to survive a primary change mid-call; xtest's own 60s
|
||||
# timeout bounds the background job regardless.
|
||||
local upgrade_log="$TOP/logs/config-upgrade-submit.out"
|
||||
: >"$upgrade_log"
|
||||
( printf 'set masters %s\nset myname cfgupgrade\nclient\nwait-primary %s\nupgradeconfig %s\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$scratch_bin" | xtest ) >"$upgrade_log" 2>&1 &
|
||||
local upgrade_pid=$!
|
||||
log "config-upgrade: submitted upgradeconfig in the background (pid=$upgrade_pid, log=$upgrade_log)"
|
||||
|
||||
local DOWN="" round PRIMARY PREV_TERM NEWP PID
|
||||
for round in $(seq 1 "$kills"); do
|
||||
log "config-upgrade: kill round $round/$kills"
|
||||
if [ -n "$DOWN" ]; then
|
||||
log "config-upgrade: restarting previously-downed $DOWN (no convergence wait — overlapping, like chaotic mode, so the upgrade genuinely interleaves with an in-progress recovery)"
|
||||
start_node "$DOWN"
|
||||
fi
|
||||
PRIMARY="$(primary_from_log)"
|
||||
if [ -z "$PRIMARY" ]; then
|
||||
wait_for_primary_log || log "config-upgrade: round $round: no primary elected yet"
|
||||
PRIMARY="$(primary_from_log)"
|
||||
fi
|
||||
if [ -z "$PRIMARY" ]; then
|
||||
log "config-upgrade: round $round: no primary to kill — skipping"
|
||||
continue
|
||||
fi
|
||||
record_primary "$PRIMARY"
|
||||
PREV_TERM="$(latest_primary_term)"; PREV_TERM="${PREV_TERM:-0}"
|
||||
PID="$(pgrep -f -- "$(node_pattern "$PRIMARY")")"
|
||||
if [ -n "$PID" ]; then
|
||||
log "config-upgrade: round $round: killing primary $PRIMARY (term=$PREV_TERM)"
|
||||
kill -9 $PID 2>/dev/null
|
||||
DOWN="$PRIMARY"
|
||||
else
|
||||
log "config-upgrade: round $round: primary $PRIMARY already gone"
|
||||
fi
|
||||
sleep 1
|
||||
NEWP="$(wait_for_new_primary "$PREV_TERM")"
|
||||
if [ -n "$NEWP" ]; then
|
||||
log "config-upgrade: round $round: new primary elected: $NEWP (term=$(latest_primary_term))"
|
||||
record_primary "$NEWP"
|
||||
else
|
||||
log "config-upgrade: round $round: no new primary within window (possible election stall)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Restore full 3-of-3 before judging anything.
|
||||
if [ -n "$DOWN" ]; then
|
||||
log "config-upgrade: restarting last-downed $DOWN and settling"
|
||||
start_node "$DOWN"
|
||||
fi
|
||||
wait_for_full_convergence \
|
||||
|| fail "config-upgrade: cluster did not reconverge to 3-of-3 after the chaotic kill sequence"
|
||||
|
||||
log "config-upgrade: waiting for the background upgrade submission to finish"
|
||||
local waited=0
|
||||
while kill -0 "$upgrade_pid" 2>/dev/null && [ "$waited" -lt 65 ]; do
|
||||
sleep 1
|
||||
waited=$((waited + 1))
|
||||
done
|
||||
wait "$upgrade_pid" 2>/dev/null
|
||||
if ! grep -q "Config upgrade submitted" "$upgrade_log"; then
|
||||
log "config-upgrade: submission did not report success while interleaved with the kills — retrying once against the now-settled, reconverged cluster (prior log: $(cat "$upgrade_log"))"
|
||||
printf 'set masters %s\nset myname cfgupgrade\nclient\nwait-primary %s\nupgradeconfig %s\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" "$scratch_bin" | xtest >"$upgrade_log" 2>&1
|
||||
grep -q "Config upgrade submitted" "$upgrade_log" \
|
||||
|| fail "config-upgrade: upgradeconfig never succeeded, even after the chaotic sequence settled: $(cat "$upgrade_log")"
|
||||
fi
|
||||
log "config-upgrade: upgrade submission confirmed committed"
|
||||
|
||||
# Give every member a moment to apply the marker and materialize its own
|
||||
# upgrade snapshot (Req 1.1/1.5 — synchronous FROM EACH MEMBER'S OWN
|
||||
# apply-pump perspective, but a behind/just-restarted member's own apply
|
||||
# can still lag the marker's commit by a few seconds).
|
||||
sleep 5
|
||||
|
||||
# 5. Materialized-state identity (Req 4.1) + job survival.
|
||||
verify_no_loss \
|
||||
|| fail "config-upgrade: a committed job was LOST across the upgrade+chaotic-kill scenario"
|
||||
|
||||
local f1 f2 f3
|
||||
f1="$(latest_snapshot_file ctrl-1)" || fail "config-upgrade: ctrl-1 has no on-disk checkpoint after the upgrade"
|
||||
f2="$(latest_snapshot_file ctrl-2)" || fail "config-upgrade: ctrl-2 has no on-disk checkpoint after the upgrade"
|
||||
f3="$(latest_snapshot_file ctrl-3)" || fail "config-upgrade: ctrl-3 has no on-disk checkpoint after the upgrade"
|
||||
log "config-upgrade: comparing latest checkpoint files: $f1 vs $f2 vs $f3"
|
||||
local sdout
|
||||
sdout="$(printf 'snapdiff %s %s\nquit\n' "$f1" "$f2" | xtest)"
|
||||
echo "$sdout" | grep -q "IDENTICAL" \
|
||||
|| fail "config-upgrade: materialized-state DIVERGENCE ctrl-1 vs ctrl-2: $sdout"
|
||||
sdout="$(printf 'snapdiff %s %s\nquit\n' "$f1" "$f3" | xtest)"
|
||||
echo "$sdout" | grep -q "IDENTICAL" \
|
||||
|| fail "config-upgrade: materialized-state DIVERGENCE ctrl-1 vs ctrl-3: $sdout"
|
||||
log "config-upgrade: materialized state IDENTICAL across all 3 members (checkpoint byte-identity)"
|
||||
|
||||
# 6. Configuration identity (Req 4.2/4.3). configcompare_paths takes each
|
||||
# member's workdir/name conf-root DIRECTORY — $TOP/controller/ctrl-N,
|
||||
# NOT .../ctrl-N/conf — it appends "/conf/current" itself
|
||||
# (configcompare.c's read_member_config_sha256; verified by reading
|
||||
# the tool's source directly before this invocation).
|
||||
local ccout
|
||||
ccout="$(printf 'configcompare %s %s %s\nquit\n' \
|
||||
"$TOP/controller/ctrl-1" "$TOP/controller/ctrl-2" "$TOP/controller/ctrl-3" | xtest)"
|
||||
echo "$ccout" | grep -q "IDENTICAL across 3 members" \
|
||||
|| fail "config-upgrade: active-CONFIGURATION divergence across members: $ccout"
|
||||
log "config-upgrade: active configuration IDENTICAL across all 3 members"
|
||||
|
||||
# 7. Host admin-status survival (Req 6.4) — checked DIRECTLY per member,
|
||||
# not inferred from the aggregate identity checks above. Rotate
|
||||
# primary through each of the 3 nodes in turn (force_elect_node
|
||||
# no-ops cheaply if the target is already primary) and assert $host
|
||||
# shows CLOSED via workerlist each time — the only way to query a
|
||||
# SPECIFIC member's own live view directly, since a standby cannot
|
||||
# answer OPCode_get (workerlist) itself.
|
||||
local n wl
|
||||
for n in "${NODES[@]}"; do
|
||||
force_elect_node "$n" || fail "config-upgrade: could not force-elect $n to directly verify its own host admin-status view"
|
||||
wl="$(printf 'set masters %s\nset myname soak\nclient\nwait-primary %s\nworkerlist\nquit\n' \
|
||||
"$MASTERS" "$WAITMS" | xtest)"
|
||||
echo "$wl" | grep -qE "^${host}[[:space:]]+CLOSED" \
|
||||
|| fail "config-upgrade: host $host is NOT admin-CLOSED while $n served as primary: $wl"
|
||||
log "config-upgrade: $host confirmed CLOSED while $n served as primary"
|
||||
done
|
||||
}
|
||||
|
||||
# ── Fact-stream primitives (raft-apply-decoupling task 7.2, Req 10.2/10.4) ───
|
||||
#
|
||||
# Drives a continuous stream of WORKER-FACT messages (jstart/jfinish/jfail —
|
||||
|
|
@ -1925,6 +2223,28 @@ if [ "$MODE" = "array-chaos-retry" ]; then
|
|||
exit 0
|
||||
fi
|
||||
|
||||
# ── Config-upgrade mode: config-upgrade-snapshot task 8.6 ────────────────────
|
||||
# Opt-in (--config-upgrade / XCE_HA_MODE=config-upgrade). See
|
||||
# run_config_upgrade_round's own header comment for the full mechanism trace.
|
||||
# Single dedicated round, not a loop — like --array-soak/--array-chaos-retry
|
||||
# (not --force-snapshot's verdict_emit-based counted-verdict style), every
|
||||
# hard invariant (job survival, materialized-state identity via snapdiff,
|
||||
# active-configuration identity via configcompare, host admin-status
|
||||
# survival) is asserted with an immediate fail() inside
|
||||
# run_config_upgrade_round() itself, so reaching the PASS line below already
|
||||
# means every check passed — no scan_bug924/V_BUG924 tally here, matching
|
||||
# array-soak/array-chaos-retry's convention (not force-snapshot/factstream's,
|
||||
# which feed V_BUG924 into a verdict_emit exclusion). Does NOT run the
|
||||
# sequential loop or leadership-coverage maneuver.
|
||||
if [ "$MODE" = "config-upgrade" ]; then
|
||||
log "MODE=config-upgrade — driving a configuration upgrade interleaved with chaotic multi-kill failover"
|
||||
run_config_upgrade_round
|
||||
log "PASS (config-upgrade): ${#ALL_JOBIDS[@]} jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members"
|
||||
cleanup
|
||||
trap - EXIT INT TERM
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Chaotic mode: overlapping-kill rounds → isolated verdict ──────────────────
|
||||
# Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up
|
||||
# windows; outcomes are COUNTED (committed divergence + committed job loss) with
|
||||
|
|
|
|||
|
|
@ -149,6 +149,12 @@
|
|||
|
||||
## Implementation Notes
|
||||
|
||||
- **Task 8.6 progress (infrastructure + two real production bugs found and fixed; task NOT YET complete — one more bug blocks a clean live-soak PASS):** building this task's live 3-node chaotic-upgrade soak required substantial new infrastructure that never existed anywhere in this codebase: a client-side send path for `OP_upgradeConfig` (`xce_upgradeconfig_req_raw`/`xce_upgradeconfig`/`xcecli_upgradeconfig` — the auto-generated capnp encoder for this request type is a literal no-op for its `Data` fields, worked around via a hand-rolled bypass mirroring the existing `xce_replconfigpush_req_raw` idiom), a `cmd_hostctrl`/`cmd_upgradeconfig` CLI pair in `test.c`, and a new `--config-upgrade` mode in `run-ha-soak.sh` (+ `SKILL.md` docs). Building and running this surfaced two real, independently-confirmed production defects:
|
||||
- **bug-1468** (`cluster.c`): `myreq_handler`'s `OPCode_upgradeConfig` dispatch guard was inverted by an unrelated commit (`fd93997f`, 2026-05-31) — it rejected the only role (primary) that could ever legitimately reach it, making config upgrades unreachable from any network client at all. Fixed: restored correct polarity.
|
||||
- **bug-1469** (`persist.c`): `apply_committed_entry`'s `WAL_CONFIG_MARKER` case never called `config_activate_committed()` — so the node that *originates* a config upgrade never refreshed its own in-memory config/queue state as part of committing it (only receiving standbys did, via a separate out-of-band push path). If that node was killed before some later, independent event happened to refresh it, its checkpoint froze with stale config content — a genuine, **live-reproduced** instance of bug-857's originally-hypothesized failure mode (task 8.5 could only prove a different, unit-level mechanism; this is the first real empirical case). Fixed: added `config_activate_committed(c)` inside the marker-apply case, positioned *before* the marker-triggered checkpoint write (checkpoint content is built from live runtime tables that only reflect the reload after activation runs, not from `myconf` directly) and guarded against `c->myconf == NULL` (a real crash risk hit in an existing test fixture). Fixing this also exposed and required fixing a genuine race in a pre-existing test (`test_commit_advance_retry_clears_push_before_commit_fence`) that the new activation side-effect made observable for the first time — narrowed the race window, did not weaken any assertion. Reviewer independently re-derived the lock-safety claim from scratch (traced every mutex site in `config_repl.c`, confirmed none reachable from `config_activate_committed`) and the thread-class claim (`apply_committed_entry` runs on a `VTASK_LIVE_BLOCK` pool worker, confirmed via `persist_apply_pump_kick`) rather than trusting the implementer's account — both held.
|
||||
- **Still open, blocking a clean PASS — bug-1471**: a separate, much narrower 1-byte checkpoint divergence in a host-admin-status frame (not a config-activation issue), found independently by two concurrent sessions working this task and root-cause-not-yet-isolated. Also noted but deliberately **not** triggered by the current soak scenario: a deeper, genuinely separate gap where no live mechanism exists anywhere in this codebase to add a *new* queue to an already-running member's runtime `qmgr` without a restart (`config_activate_committed`/`load_primary_config` refresh `myconf->cluster_config` but nothing syncs that into the live `qmgr`) — the soak's upgrade-target config change was deliberately scoped to modifying an *existing* queue's field rather than adding one, specifically to test what Requirements 4.1-4.3/6.1/6.4 actually ask for without tripping over this separate, deeper architectural gap. Both are flagged in `.wolf/buglog.json` for whoever next picks up bug-1471 (or a dedicated follow-up for the queue-creation gap) — task 8.6's own checkbox stays unchecked until the soak reports a clean PASS.
|
||||
- **Process note**: this task was interrupted by the same monthly-spend-limit condition hit earlier in this session (task 6.1), across three separate dispatch/resume cycles. Each resumption correctly checked live `git status`/`diff` before acting rather than blindly redoing work, and the final state was independently re-verified end-to-end (build + full 1025-test suite + file-integrity spot check) before being reviewed and committed — no corruption or duplicated work resulted, but it's worth flagging that this class of interruption can leave multiple agent instances transiently active in the same worktree.
|
||||
|
||||
- **Task 8.5 redirection (design.md's original mechanism was wrong; this is not the same as a review rejection):** the original hypothesis for bug-857's divergence — a job whose queue is removed by a config upgrade gets re-homed to `lost_and_found` on some members but not others, based on per-node restart timing — turned out to be **aspirational, never-implemented code**. Exhaustive grep across `job.c`/`queue.c`/`worker.c`/`config_repl.c` at both the pre-fix baseline commit (`e0c5e8b7`) and current HEAD confirmed: `create_lost_and_found` only ever creates the sentinel queue/host objects (and its entire body is `#ifndef XCE_TESTING`-gated); `restore_job`/`restore_job_array` never consult `qmgr`; `restore_queue_state` explicitly *skips* rather than re-homes when a queue is missing. `.wolf/buglog.json`'s bug-857 entry itself flags this: "identified via TLA+ gap analysis... NOT empirically reproduced yet." design.md's Requirements Traceability table assigned 6.2 to a live "HA soak new scenario" — also written against the same since-disproven hypothesis. Redirected (orchestrator decision, not a task-local implementer call) to a real, code-backed mechanism instead: `schedule_checkpoint`'s pre-existing, unmodified-by-this-spec WAL-record-count threshold trigger is genuinely decoupled from the config-marker `(term,lsn)` boundary, so two members' independently-timed checkpoints can capture different job/queue sets — the actual, general-purpose truth behind bug-857's claim ("no snapshot transfers an identical materialized state"). This is testable entirely within `controller_unit_test`, no live multi-process soak or old-commit worktree needed, since the threshold-trigger mechanism is still live, unmodified production code coexisting with the new marker-trigger at HEAD. Landed as `test_bug857_regression.c` (2 tests: divergence under threshold-only triggering vs. convergence under the real marker-triggered path, the latter under deliberately differing per-member processing pace to make the convergence claim non-trivial). Reviewer independently re-derived and confirmed the mechanism claim from first principles (not just re-running the tests) before approving — see `git blame`/review transcript for the full verification chain (schedule_checkpoint predates this spec via `git merge-base --is-ancestor`, zero lines of its body touched by tasks 1-7, `persist_apply_pump_run`'s strictly-sequential dispatch loop explains why the marker-triggered checkpoint's convergence isn't a timing coincidence). design.md's Requirements Traceability table (row 6.2) and Testing Strategy section were both corrected to match. **Heads-up for tasks 8.6-8.9**: they still legitimately need live multi-process HA-soak-level testing for Requirement 6.1 (chaotic multi-kill + config upgrade) — that infrastructure (a live config-upgrade CLI trigger doesn't exist anywhere in this codebase yet; confirmed via grep of `test.c`/`src/frontend`/`src/tools`) was NOT built by this task and will need to be built by whichever of 8.6-8.9 first needs to drive a live `OP_upgradeConfig` RPC against a running cluster.
|
||||
|
||||
- **Task 8.4 round-1 rejection (real gap, caught by review):** the first pass built `configcompare_paths()` (correct, well-tested against `persist.c`/`config_repl.c`'s real `workdir/name/conf/current`+`conf/cluster.bin.<hex>` resolution convention) and linked `configcompare.c` into both `xce_test_src` and `controller_unit_test_src` in meson, but never registered it as a `cmd_configcompare` subcommand in `testsuite/test.c` — unlike its siblings `walcompare`/`snapverify`/`snapdiff`, each of which has a `cmd_*` wrapper in BOTH `top_commands[]` and `client_commands[]`. Left as delivered, the tool was a tested library function with no shell entry point — unusable by task 8.6/8.9's later HA soak scenarios, which design.md explicitly says will invoke `configcompare` directly against each node's data directory. Fixed with a 25-line, `test.c`-only remediation mirroring `cmd_snapdiff`'s exact shape; verified end-to-end via `echo 'configcompare <dir1> <dir2>' | xce_test` (checked `$?` for both IDENTICAL/exit-0 and DIVERGENT/exit-1 cases, not just printed text). **Heads-up for any future task in this spec (or elsewhere) that adds a new offline testsuite tool in this family**: linking the `.c` file into the meson source lists is NOT sufficient — it must also get a `cmd_*` wrapper + `top_commands[]`/`client_commands[]` entries in `test.c`, or it silently has no way to be invoked outside the unit test binary.
|
||||
|
|
|
|||
|
|
@ -669,10 +669,30 @@ static int myreq_handler(vtask_t* task)
|
|||
break;
|
||||
case OPCode_upgradeConfig:
|
||||
#if CONF_ENABLE_UPGRADE_CONFIG
|
||||
/* OPCode_upgradeConfig is sent BY the primary TO standbys to push config.
|
||||
* Standbys must apply it; only a primary sending this to itself is invalid. */
|
||||
if (ha_is_primary(&c->ha)) {
|
||||
/* Primary should not receive its own config push — ignore */
|
||||
/* OPCode_upgradeConfig is sent BY an admin client (or xceadmin's
|
||||
* config-push tool) TO the primary controller, which decodes the
|
||||
* blob, verifies it, and drives it through config_commit_change()
|
||||
* (the (term,lsn) marker consensus path) — see
|
||||
* do_upgrade_config_with_bytes's doc comment, config_repl.c. Only the
|
||||
* primary can commit a WAL entry, so a standby cannot service this
|
||||
* request; ha_dispatch_gate already denies OPCode_upgradeConfig for
|
||||
* every non-primary role (it is absent from the CTRL_STANDBY /
|
||||
* CTRL_INIT / CTRL_ELECTING allow-lists, ha.c), so this check is a
|
||||
* defensive belt-and-suspenders guard, not the primary gate.
|
||||
*
|
||||
* bug-fix (2026-07-25): this check was inverted from
|
||||
* commit fd93997f ("election-iomgr-refactor", 2026-05-31) onward —
|
||||
* it rejected the request WHEN primary, unconditionally reachable
|
||||
* only on primary (per the dispatch-gate note above), making
|
||||
* handle_upgrade_config() permanently unreachable from any network
|
||||
* caller (only unit tests calling it directly ever exercised it).
|
||||
* Independently diagnosed and documented as a blocking risk by the
|
||||
* admin-config-tool spec (design.md "Boundary Commitments" /
|
||||
* "Risks") before config-upgrade-snapshot task 8.6 needed a live
|
||||
* OP_upgradeConfig sender and hit the same wall. Restored to the
|
||||
* pre-regression (correct) polarity. */
|
||||
if (!ha_is_primary(&c->ha)) {
|
||||
/* Standby cannot commit a config change — reject loud. */
|
||||
xce_reply_t ureply;
|
||||
init_reply_by_req(&ureply, &request);
|
||||
ureply.status = -EAGAIN;
|
||||
|
|
|
|||
|
|
@ -56,6 +56,20 @@ extern int restore_host_state(cluster_t* c, const char* name,
|
|||
* initialized default. */
|
||||
extern int upgrade_materialization_verify(cluster_t* c);
|
||||
|
||||
/* config_activate_committed defined in config_repl.c (config-upgrade-snapshot
|
||||
* task 5/6.2 remediation, bug-1469, Req 1.8/3.2/3.4) -- re-parses the
|
||||
* locally-held config bytes referenced by the latest committed
|
||||
* WAL_CONFIG_MARKER into c->myconf (load_primary_config), hash-verifying
|
||||
* them first. Called unconditionally from apply_committed_entry's
|
||||
* WAL_CONFIG_MARKER case below, BEFORE upgrade_materialization_verify: every
|
||||
* member's own live apply of a marker it just committed must refresh its
|
||||
* own c->myconf, not only members that separately receive an out-of-band
|
||||
* handle_repl_config_push (see the call site's own doc comment for the full
|
||||
* rationale). Declared here via extern, mirroring upgrade_materialization_
|
||||
* verify just above, rather than pulling in config_repl.h's full include
|
||||
* surface for one symbol. */
|
||||
extern int config_activate_committed(cluster_t* c);
|
||||
|
||||
/* raft-apply-decoupling task 3.2: pending command-apply hand-off list node
|
||||
* (producer side; the not-yet-built ordered apply pump, task 6.1/6.2, is
|
||||
* the consumer). Declared here (rather than persist.h) so the list stays
|
||||
|
|
@ -3807,6 +3821,102 @@ int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn,
|
|||
* checkpoint. `lsn` (this function's own parameter) is
|
||||
* already the marker's exact position, so it is passed
|
||||
* through explicitly instead of re-derived. */
|
||||
/* bug-1469 fix (live HA soak, config-upgrade-snapshot task 8.6):
|
||||
* unconditionally re-run config activation right here, BEFORE the
|
||||
* forced checkpoint below, so EVERY member -- primary, standby, and
|
||||
* any future newly-elected leader -- is guaranteed to have
|
||||
* refreshed c->myconf (and the runtime queue/host tables
|
||||
* load_primary_config provisions from it) from disk BEFORE the
|
||||
* checkpoint writer walks that same runtime state to build the
|
||||
* queue-entity list. Before this call existed at all,
|
||||
* config_activate_committed() was reachable from exactly two
|
||||
* places: handle_repl_config_push (a STANDBY receiving a push) and
|
||||
* the boot/recovery path -- never from a member's own live apply of
|
||||
* the marker it just committed. A live 3-node chaotic-kill soak
|
||||
* caught the gap empirically: the primary that ORIGINATED a config
|
||||
* upgrade (do_upgrade_config_with_bytes, which saves the new bytes
|
||||
* to disk and bumps config_generation but never re-activates them
|
||||
* into c->myconf on itself) forced a checkpoint at the marker's own
|
||||
* LSN whose queue-entity section still reflected the OLD
|
||||
* (pre-upgrade) queue set, then was killed before any LATER,
|
||||
* independent event (restart, or a racing inbound push) happened to
|
||||
* call config_activate_committed -- freezing a stale
|
||||
* queue-entity list into that checkpoint forever (a real
|
||||
* cross-member materialized-state divergence,
|
||||
* .wolf/buglog.json bug-1469's byte-level trace: the checkpoint's
|
||||
* embedded raw config-bytes section correctly carried the new
|
||||
* bytes, but the SEPARATE queue-entity list section -- built by
|
||||
* walking runtime state, not the raw bytes -- did not, because that
|
||||
* runtime state was never refreshed before the write).
|
||||
*
|
||||
* MUST run BEFORE persist_checkpoint_marker_triggered(), not merely
|
||||
* before upgrade_materialization_verify: the checkpoint writer
|
||||
* (write_snapshot -> collect_queue_frames/collect_host_frames)
|
||||
* builds its queue/host-entity sections by walking c->qmgr/
|
||||
* c->workermgr -- LIVE RUNTIME TABLES, populated at boot from
|
||||
* c->myconf->cluster_config and never otherwise re-synced from a
|
||||
* later config change on this call path -- not by re-parsing
|
||||
* mc->cluster_config directly. So activation must settle BEFORE
|
||||
* persist_checkpoint_marker_triggered() runs, or the checkpoint
|
||||
* this marker forces still captures whatever runtime state existed
|
||||
* prior to activation, regardless of how soon afterward activation
|
||||
* runs. (This ordering constraint is logically necessary given
|
||||
* write_snapshot's own read path, confirmed by reading
|
||||
* collect_queue_frames/write_snapshot directly -- not yet
|
||||
* separately confirmed against a live soak re-run at the time this
|
||||
* comment was written; see the task's own final live-soak
|
||||
* verification step for that.)
|
||||
*
|
||||
* Symmetric with task 6.2's own fix shape below: one unconditional
|
||||
* call at the single apply seam every member's marker-apply is
|
||||
* guaranteed to pass through, rather than patching only the
|
||||
* originating code path.
|
||||
*
|
||||
* Safe to call here for the same reason task 6.2's own call
|
||||
* (upgrade_materialization_verify, below) is safe at this exact
|
||||
* seam: neither config_activate_committed nor anything it
|
||||
* transitively calls (config_committed_marker, local_config_sha256,
|
||||
* conf_read_current_filename, conf_read_raw, load_primary_config)
|
||||
* acquires apply_pump_lock, p->apply_pump_lock, or any other lock
|
||||
* (verified -- config_repl.c's only two mutexes, c->ha.lock and
|
||||
* c->persist.lock, are both taken in unrelated functions never on
|
||||
* this call path) -- so no reentrant-lock hazard even though this
|
||||
* runs while persist_apply_pump_run holds apply_pump_lock across
|
||||
* its entire dispatch. The disk I/O this performs (WAL segment
|
||||
* scan, config file read, Cap'n Proto decode) is the same class
|
||||
* already unconditionally performed at this exact call site by
|
||||
* upgrade_materialization_verify below (config_committed_marker's
|
||||
* own scan + a config-bytes read/hash), so this adds no new
|
||||
* blocking-I/O hazard class. Return value intentionally ignored
|
||||
* (mirrors upgrade_materialization_verify below): a
|
||||
* mismatch/absence return (-EAGAIN/-ENOENT) means this member's own
|
||||
* serve-fence (config_serve_fenced, a SEPARATE mechanism from the
|
||||
* Activation Fence below) is set instead, and that fence already
|
||||
* has its own established retry path (config_schedule_activate_retry)
|
||||
* driven off commit-advance -- this call's only job here is to give
|
||||
* every marker-apply one unconditional attempt so a member never
|
||||
* depends solely on a later, independent event to activate.
|
||||
*
|
||||
* Guarded on c->myconf != NULL: in production myconf is always
|
||||
* wired during cluster init, well before any commit/apply can run,
|
||||
* so this is a no-op check on every real code path -- but
|
||||
* config_activate_committed's own -ENOENT fallback calls
|
||||
* load_primary_config(c) unconditionally on c->myconf without its
|
||||
* own null check (pre-existing in config_repl.c, out of this fix's
|
||||
* scope to touch), and this call site is reachable from a
|
||||
* bare/otherwise-empty test cluster_t that exercises
|
||||
* WAL_CONFIG_MARKER apply mechanics without wiring myconf
|
||||
* (test_upgrade_snapshot_apply.c's tusa_t fixture,
|
||||
* test_marker_apply_via_live_pump_checkpoints_at_marker_own_lsn --
|
||||
* confirmed via a live SEGV in load_primary_config, address 0x28,
|
||||
* before this guard was added) -- mirrors this same file's existing
|
||||
* tolerance of a bare fixture elsewhere (e.g. collect_queue_frames'
|
||||
* `c->qmgr == NULL` check, read_active_config_bytes' "no live
|
||||
* cluster back-pointer" handling) rather than crashing a test that
|
||||
* predates and is unrelated to config activation. */
|
||||
if (c->myconf != NULL)
|
||||
(void)config_activate_committed(c);
|
||||
|
||||
int rc;
|
||||
#ifdef XCE_TESTING
|
||||
if (persist_marker_checkpoint_fault_hook != NULL &&
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@
|
|||
|
||||
#include <cjson/cJSON.h>
|
||||
#include <cjson/s2j.h>
|
||||
#include <openssl/sha.h> /* SHA256() — cmd_upgradeconfig hashes the config blob */
|
||||
|
||||
#include "client.h"
|
||||
#include "config.h" /* XCE_MAXNAME_LEN */
|
||||
|
|
@ -122,6 +123,8 @@ extern int cmd_resval(int, char*[]);
|
|||
extern int cmd_jobkill(int, char*[]);
|
||||
extern int cmd_jobckpt(int, char*[]);
|
||||
extern int cmd_qctrl(int, char*[]);
|
||||
extern int cmd_hostctrl(int, char*[]);
|
||||
extern int cmd_upgradeconfig(int, char*[]);
|
||||
extern int cmd_runtasks(int, char*[]);
|
||||
extern int cmd_setpend(int, char*[]);
|
||||
extern int cmd_jobinfo(int, char*[]);
|
||||
|
|
@ -230,6 +233,9 @@ static command_t client_commands[] = {
|
|||
{ "reslist", cmd_reslist, "Display cluster resources" },
|
||||
{ "workerlist", cmd_workerlist, "list workers" },
|
||||
{ "qctrl", cmd_qctrl, "queue control" },
|
||||
{ "hostctrl", cmd_hostctrl, "host control" },
|
||||
{ "upgradeconfig", cmd_upgradeconfig,
|
||||
"submit a config upgrade (cluster.bin file)" },
|
||||
{ "joblist", cmd_joblist, "list jobs" },
|
||||
{ "expect-jobs", cmd_expect_jobs, "assert current job count" },
|
||||
{ "expect-job", cmd_expect_job, "assert a JobId is present" },
|
||||
|
|
@ -1020,7 +1026,15 @@ int cmd_reslist(int argc, char* argv[])
|
|||
|
||||
static char* worker_status_str(uint32_t status)
|
||||
{
|
||||
if ((status & XCE_WORKER_UNAVAIL) != 0) {
|
||||
/* Admin-set CLOSED takes priority over the dynamic UNAVAIL bit: the
|
||||
* config-upgrade-snapshot soak (task 8.6) needs to assert a host's
|
||||
* administratively-set status specifically, and a host closed via
|
||||
* hostctrl that later disconnects (e.g. its xce_test `worker`/`join`
|
||||
* connection exits after the script ends) would otherwise print only
|
||||
* "UNAVAIL", masking the admin bit this check exists to verify. */
|
||||
if ((status & XCE_WORKER_CLOSED) != 0) {
|
||||
return "CLOSED";
|
||||
} else if ((status & XCE_WORKER_UNAVAIL) != 0) {
|
||||
return "UNAVAIL";
|
||||
} else if ((status & XCE_WORKER_LOCKEDU) != 0) {
|
||||
return "LOCKU";
|
||||
|
|
@ -2224,6 +2238,139 @@ int cmd_qctrl(int argc, char* argv[])
|
|||
return 0;
|
||||
}
|
||||
|
||||
/* hostctrl -o|-c -h <hostname> — administratively open/close a host
|
||||
* (xce_hostctrl_t{action, hostname} via xcecli_hostctrl). Mirrors cmd_qctrl's
|
||||
* exact getopt shape; only -o/-c are wired (matching xce_hostctrl_req's own
|
||||
* accepted action range, request.c — HOST_TAG/HOST_UNTAG exist in the enum
|
||||
* but are rejected -EINVAL server-side today, so no flag is offered for them
|
||||
* here). Used by the config-upgrade chaotic soak (task 8.6) to set a host's
|
||||
* admin status before/around a config upgrade and assert it survives. */
|
||||
int cmd_hostctrl(int argc, char* argv[])
|
||||
{
|
||||
int opt;
|
||||
int rc;
|
||||
xce_hostctrl_t hostctrl;
|
||||
int action = HOST_DO_NONE;
|
||||
char* hostname = NULL;
|
||||
|
||||
optind = 1;
|
||||
|
||||
while ((opt = getopt(argc, argv, "och:")) != -1) {
|
||||
switch (opt) {
|
||||
case 'o':
|
||||
action = HOST_OPEN;
|
||||
break;
|
||||
case 'c':
|
||||
action = HOST_CLOSE;
|
||||
break;
|
||||
case 'h':
|
||||
hostname = strdup(optarg);
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (hostname == NULL) {
|
||||
fprintf(stderr, "you need to specify host name with -h\n");
|
||||
return -1;
|
||||
}
|
||||
|
||||
if (action == HOST_DO_NONE) {
|
||||
fprintf(stderr, "you need to specify -o (open) or -c (close)\n");
|
||||
free(hostname);
|
||||
return -1;
|
||||
}
|
||||
|
||||
hostctrl.action = action;
|
||||
hostctrl.hostname = hostname;
|
||||
|
||||
rc = xcecli_hostctrl(active_ctx->ctx, &hostctrl);
|
||||
if (rc < 0) {
|
||||
perror("xcecli_hostctrl");
|
||||
free(hostname);
|
||||
return rc;
|
||||
}
|
||||
|
||||
printf("Operation on host %s done\n", hostname);
|
||||
|
||||
free(hostname);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* upgradeconfig <cluster.bin> — submit a configuration upgrade
|
||||
* (config-upgrade-snapshot task 8.6). Reads the given capnp-encoded
|
||||
* ClusterConfig blob (the same format `xceadmin config build` produces and
|
||||
* deploy-ha.sh installs as cluster.bin), computes its SHA-256 (OpenSSL
|
||||
* SHA256(), matching config_repl.c's do_upgrade_config_with_bytes transit
|
||||
* verification), and drives it through xcecli_upgradeconfig ->
|
||||
* xce_upgradeconfig_req_raw (OP_upgradeConfig). Mirrors cmd_snapdiff's
|
||||
* bounds-checked file-reading style. */
|
||||
int cmd_upgradeconfig(int argc, char* argv[])
|
||||
{
|
||||
if (argc != 2) {
|
||||
fprintf(stderr, "usage: upgradeconfig <cluster.bin>\n");
|
||||
return -1;
|
||||
}
|
||||
|
||||
const char* path = argv[1];
|
||||
FILE* fp = fopen(path, "rb");
|
||||
if (fp == NULL) {
|
||||
fprintf(stderr, "upgradeconfig: cannot open %s: %s\n", path,
|
||||
strerror(errno));
|
||||
return -1;
|
||||
}
|
||||
|
||||
if (fseek(fp, 0, SEEK_END) != 0) {
|
||||
fprintf(stderr, "upgradeconfig: fseek failed on %s: %s\n", path,
|
||||
strerror(errno));
|
||||
fclose(fp);
|
||||
return -1;
|
||||
}
|
||||
|
||||
long fsz = ftell(fp);
|
||||
if (fsz <= 0) {
|
||||
fprintf(stderr, "upgradeconfig: %s is empty or ftell failed\n", path);
|
||||
fclose(fp);
|
||||
return -1;
|
||||
}
|
||||
rewind(fp);
|
||||
|
||||
uint8_t* bytes = (uint8_t*)malloc((size_t)fsz);
|
||||
if (bytes == NULL) {
|
||||
fprintf(stderr, "upgradeconfig: out of memory reading %s\n", path);
|
||||
fclose(fp);
|
||||
return -1;
|
||||
}
|
||||
|
||||
size_t nread = fread(bytes, 1, (size_t)fsz, fp);
|
||||
fclose(fp);
|
||||
if (nread != (size_t)fsz) {
|
||||
fprintf(stderr, "upgradeconfig: short read on %s (%zu of %ld bytes)\n",
|
||||
path, nread, fsz);
|
||||
free(bytes);
|
||||
return -1;
|
||||
}
|
||||
|
||||
uint8_t sha256[SHA256_DIGEST_LENGTH];
|
||||
SHA256(bytes, (size_t)fsz, sha256);
|
||||
|
||||
int rc = xcecli_upgradeconfig(active_ctx->ctx, sha256,
|
||||
sizeof(sha256), bytes, (size_t)fsz);
|
||||
free(bytes);
|
||||
|
||||
if (rc < 0) {
|
||||
fprintf(stderr, "upgradeconfig: failed (rc=%d): %s\n", rc,
|
||||
strerror(-rc));
|
||||
return rc;
|
||||
}
|
||||
|
||||
printf("Config upgrade submitted from %s (%ld bytes)\n", path, fsz);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int cmd_runtasks(int argc, char* argv[])
|
||||
{
|
||||
xce_taskhost_t hosts[MAX_TASK_HOSTS];
|
||||
|
|
|
|||
|
|
@ -1819,9 +1819,25 @@ static void test_commit_advance_retry_clears_push_before_commit_fence(void)
|
|||
uint64_t tagN; { uint64_t t; memcpy(&t, shaN, 8); tagN = be64toh(t); }
|
||||
TEST_ASSERT_FALSE(memcmp(shaO, shaN, 32) == 0);
|
||||
|
||||
/* The NEW marker is APPENDED + flushed to the WAL (local tip advances). */
|
||||
/* The NEW marker is APPENDED + flushed to the WAL (local tip advances).
|
||||
* bug-1469 fix interaction: flushing makes durable_lsn advance, and this
|
||||
* single-member fixture's commit engine (majority=1) reacts to ANY
|
||||
* durable advance by racing committed_lsn forward on its OWN background
|
||||
* fsync-worker thread (commit_on_durable_advance) -- which, since
|
||||
* apply_committed_entry's WAL_CONFIG_MARKER case now ALSO activates
|
||||
* config (bug-1469), could apply-and-activate this NEW marker for real
|
||||
* before this test's own thread gets a chance to pin committed_lsn back
|
||||
* down below it. PIN IMMEDIATELY, right here, before any other work
|
||||
* (previously this pin lived after conf_save_config below, leaving a
|
||||
* multi-syscall window — several real fsyncs — for that background race
|
||||
* to win against a single-member fixture's over-eager auto-commit,
|
||||
* something a real multi-node standby never does since its
|
||||
* committed_lsn is solely driven by the primary's commit-advance, never
|
||||
* local durability). */
|
||||
uint64_t new_mlsn = t43_append_marker_uncommitted(c, tagN, shaN);
|
||||
TEST_ASSERT_TRUE(new_mlsn > committed_at_old);
|
||||
atomic_store_explicit(&c->persist.committed_lsn, committed_at_old,
|
||||
memory_order_release);
|
||||
|
||||
/* The OLD config is currently LIVE (activated in upgrade 1). Capture the live
|
||||
* decoded-config pointer as the activation witness: load_primary_config frees
|
||||
|
|
@ -1837,9 +1853,10 @@ static void test_commit_advance_retry_clears_push_before_commit_fence(void)
|
|||
* commit barrier, which would advance committed_lsn to the durable WAL tip, and
|
||||
* it bumps the advertised config_version_tag to tagN (this is the very reason
|
||||
* the primary's config-behind re-push net does not re-fire — Finding D); we
|
||||
* save FIRST, then PIN committed_lsn back below the NEW marker to model a
|
||||
* standby whose committed prefix (driven by the primary's commit-advance, not
|
||||
* local durability) has NOT yet advanced past the NEW marker. */
|
||||
* already pinned committed_lsn back below the NEW marker immediately above (to
|
||||
* model a standby whose committed prefix, driven by the primary's commit-
|
||||
* advance, not local durability, has NOT yet advanced past the NEW marker) —
|
||||
* re-pin again here for defense-in-depth against the same race. */
|
||||
TEST_ASSERT_EQUAL_INT(0, conf_save_config(mc, bytesN, blenN, shaN));
|
||||
atomic_store_explicit(&c->persist.committed_lsn, committed_at_old,
|
||||
memory_order_release);
|
||||
|
|
@ -2329,6 +2346,147 @@ static void test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss(voi
|
|||
printf("test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss: PASS\n");
|
||||
}
|
||||
|
||||
/* ── bug-1469 regression: WAL_CONFIG_MARKER apply must ACTIVATE config on
|
||||
* the ORIGINATING member, not just receiving standbys ──────────────────────
|
||||
*
|
||||
* Live 3-node chaotic-kill soak (config-upgrade-snapshot task 8.6,
|
||||
* .wolf/buglog.json bug-1469): a config upgrade's WAL_CONFIG_MARKER commits
|
||||
* and even checkpoints successfully on the ORIGINATING member (the primary
|
||||
* that submitted the upgrade via do_upgrade_config_with_bytes), but
|
||||
* apply_committed_entry's WAL_CONFIG_MARKER case (persist.c) never called
|
||||
* config_activate_committed()/load_primary_config() -- so that member's
|
||||
* c->myconf stayed parsed from the OLD config bytes until some LATER,
|
||||
* independent event (a restart, or an incoming out-of-band
|
||||
* handle_repl_config_push) happened to refresh it. The live soak killed the
|
||||
* originating primary shortly after it committed the upgrade, before any
|
||||
* such later event, freezing a stale c->myconf into its last checkpoint --
|
||||
* a genuine cross-member materialized-state divergence (ctrl-1's checkpoint
|
||||
* ended up a different byte size than ctrl-2/ctrl-3's, both LSN 11).
|
||||
* config_activate_committed WAS already being called correctly on
|
||||
* RECEIVING standbys, via handle_repl_config_push -- this gap is specific
|
||||
* to the member that originates/commits the marker on itself.
|
||||
*/
|
||||
|
||||
/* Build a valid ClusterConfig blob distinguishable by config_generation +
|
||||
* clustername (mirrors t43_make_config_blob, parameterized so this test can
|
||||
* commit two genuinely different, independently-decodable configs -- an
|
||||
* "old" pre-upgrade config and a "new" post-upgrade config -- without
|
||||
* relying on a byte-mutation of an opaque capnp blob, which risks
|
||||
* corrupting the encoding; config-upgrade-snapshot's own standing gotcha
|
||||
* notes t43_make_config_blob() returns byte-IDENTICAL content on repeat
|
||||
* calls, so a genuinely different SECOND blob needs its own encode call). */
|
||||
static uint8_t *t1469_make_config_blob(uint64_t gen, const char *name,
|
||||
size_t *len_out)
|
||||
{
|
||||
cluster_config_t cfg;
|
||||
struct capn cap;
|
||||
struct capn_segment *cs;
|
||||
ClusterConfig_ptr p;
|
||||
uint8_t *buf;
|
||||
size_t sz;
|
||||
|
||||
memset(&cfg, 0, sizeof(cfg));
|
||||
cfg.clustername = (char *)name;
|
||||
cfg.config_generation = gen;
|
||||
|
||||
capn_init_malloc(&cap);
|
||||
cs = capn_root(&cap).seg;
|
||||
encode_ClusterConfig_ptr(cs, &p, &cfg);
|
||||
capn_setp(capn_root(&cap), 0, p.p);
|
||||
sz = (size_t)capn_size(&cap);
|
||||
buf = (uint8_t *)malloc(sz);
|
||||
TEST_ASSERT_NOT_NULL(buf);
|
||||
capn_write_mem(&cap, buf, sz, 0);
|
||||
capn_free(&cap);
|
||||
*len_out = sz;
|
||||
return buf;
|
||||
}
|
||||
|
||||
/* This test drives the upgrade marker through the REAL production apply
|
||||
* path (config_append_marker + commit_submit + wait-for-applied, exactly
|
||||
* like test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss
|
||||
* above) on a single-member fixture acting as its OWN primary -- i.e. the
|
||||
* ORIGINATING member applying its own committed marker, never receiving an
|
||||
* out-of-band handle_repl_config_push. It NEVER calls
|
||||
* config_activate_committed() for the upgrade marker itself, so the only
|
||||
* way mc->cluster_config can reflect the new config is via
|
||||
* apply_committed_entry's own WAL_CONFIG_MARKER dispatch.
|
||||
*
|
||||
* RED (before the fix): mc->cluster_config stays pinned to the OLD config
|
||||
* (config_generation==1, "t1469-old") after the new marker's real,
|
||||
* successful commit+apply -- proving the marker-triggered dispatch never
|
||||
* activates config on the member that originated it.
|
||||
* GREEN (after the fix): mc->cluster_config reflects the NEW config
|
||||
* (config_generation==2, "t1469-new") immediately after the marker's apply
|
||||
* returns, with no restart and no out-of-band push. */
|
||||
static void test_wal_config_marker_apply_activates_originating_member_config(void)
|
||||
{
|
||||
char tmp[256];
|
||||
strcpy(tmp, "/tmp/xce_cr1469_activate_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||
|
||||
cluster_t *c = t42_make_cluster(tmp);
|
||||
TEST_ASSERT_NOT_NULL(c);
|
||||
t42_make_leader(c, /*term*/69, /*n_peers*/1);
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
commit_engine_init(&c->commit, &c->persist, 1, 8));
|
||||
repl_set_commit_engine(&c->persist.repl, &c->commit);
|
||||
myconf_t *mc = t43_attach_myconf(c, tmp);
|
||||
|
||||
/* Establish the pre-upgrade steady state: an OLD config already
|
||||
* committed and ACTIVATED via the legitimate, already-tested
|
||||
* config_activate_committed path (mirrors 4.3-a above) -- this member
|
||||
* is already running under config_generation=1 before any upgrade is
|
||||
* ever submitted, exactly like a real cluster member. */
|
||||
size_t oldlen = 0;
|
||||
uint8_t *oldbytes = t1469_make_config_blob(1, "t1469-old", &oldlen);
|
||||
uint8_t oldsha[32];
|
||||
SHA256(oldbytes, oldlen, oldsha);
|
||||
uint64_t oldtag; { uint64_t t; memcpy(&t, oldsha, 8); oldtag = be64toh(t); }
|
||||
TEST_ASSERT_EQUAL_INT(0, conf_save_config(mc, oldbytes, oldlen, oldsha));
|
||||
free(oldbytes);
|
||||
t43_commit_marker(c, oldtag, oldsha);
|
||||
TEST_ASSERT_EQUAL_INT(0, config_activate_committed(c));
|
||||
TEST_ASSERT_NOT_NULL(mc->cluster_config);
|
||||
TEST_ASSERT_EQUAL_UINT64(1, mc->cluster_config->config_generation);
|
||||
|
||||
/* Now the upgrade: NEW config bytes saved locally exactly as
|
||||
* do_upgrade_config_with_bytes's own step 5 (conf_save_config) does on
|
||||
* the ORIGINATING primary, then a NEW marker committed through the
|
||||
* exact same real path bug-1469's live soak exercised. Deliberately
|
||||
* NEVER calls config_activate_committed() for this marker anywhere
|
||||
* below -- activation must come from apply_committed_entry's own
|
||||
* dispatch, or this assertion fails. */
|
||||
size_t newlen = 0;
|
||||
uint8_t *newbytes = t1469_make_config_blob(2, "t1469-new", &newlen);
|
||||
uint8_t newsha[32];
|
||||
SHA256(newbytes, newlen, newsha);
|
||||
uint64_t newtag; { uint64_t t; memcpy(&t, newsha, 8); newtag = be64toh(t); }
|
||||
TEST_ASSERT_EQUAL_INT(0, conf_save_config(mc, newbytes, newlen, newsha));
|
||||
free(newbytes);
|
||||
t43_commit_marker(c, newtag, newsha);
|
||||
|
||||
/* THE ASSERTION THAT PROVES THE FIX: mc->cluster_config must now
|
||||
* reflect the NEW config -- purely from apply_committed_entry's own
|
||||
* WAL_CONFIG_MARKER dispatch, exactly the production path the
|
||||
* ORIGINATING primary runs on itself. Before the fix this stays pinned
|
||||
* at generation 1 ("t1469-old") forever -- exactly bug-1469's observed
|
||||
* cross-member divergence, reproduced at the unit level. */
|
||||
TEST_ASSERT_NOT_NULL(mc->cluster_config);
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(2, mc->cluster_config->config_generation,
|
||||
"bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never "
|
||||
"activated the new config on the ORIGINATING member -- c->myconf "
|
||||
"stayed pinned to the OLD config");
|
||||
TEST_ASSERT_EQUAL_STRING("t1469-new", mc->cluster_config->clustername);
|
||||
TEST_ASSERT_EQUAL_HEX64(newtag, mc->config_version_tag);
|
||||
TEST_ASSERT_EQUAL_INT(0, config_serve_fenced(c));
|
||||
|
||||
t43_free_myconf(c);
|
||||
commit_engine_fini(&c->commit);
|
||||
t42_teardown(c);
|
||||
printf("test_wal_config_marker_apply_activates_originating_member_config: PASS\n");
|
||||
}
|
||||
|
||||
/* ── wait_all_with_timeout (XCE_TESTING shim) ───── */
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
|
|
@ -5144,6 +5302,9 @@ void test_config_repl(void)
|
|||
* production trigger (apply_committed_entry's WAL_CONFIG_MARKER case),
|
||||
* not a direct test call -- proves the fence is reachable at all. */
|
||||
RUN_TEST(test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss);
|
||||
/* bug-1469: WAL_CONFIG_MARKER apply must activate config on the
|
||||
* ORIGINATING member too, not just receiving standbys. */
|
||||
RUN_TEST(test_wal_config_marker_apply_activates_originating_member_config);
|
||||
#ifdef XCE_TESTING
|
||||
RUN_TEST(test_load_primary_config_missing_returns_error);
|
||||
RUN_TEST(test_wait_all_with_timeout_slow_threads);
|
||||
|
|
|
|||
|
|
@ -95,6 +95,14 @@ extern int xcecli_jobctrl(xcecli_ctx_t, xce_jobctrl_t*);
|
|||
extern int xcecli_notify_jobchkpt(xcecli_ctx_t, uint64_t, char*);
|
||||
extern int xcecli_queuectrl(xcecli_ctx_t, xce_queuectrl_t*);
|
||||
extern int xcecli_hostctrl(xcecli_ctx_t, xce_hostctrl_t*);
|
||||
/* xcecli_upgradeconfig: submit a configuration upgrade (sha256 + serialised
|
||||
* ClusterConfig blob) to the primary via OP_upgradeConfig, transparently
|
||||
* surviving a primary change mid-call via XCECLI_FAILOVER_CALL (same
|
||||
* write-op failover contract as xcecli_hostctrl/xcecli_queuectrl). Returns
|
||||
* the server's reply status (0 on success). */
|
||||
extern int xcecli_upgradeconfig(xcecli_ctx_t, const uint8_t *sha256,
|
||||
size_t sha256_len, const uint8_t *bytes,
|
||||
size_t bytes_len);
|
||||
extern int xcecli_runtask(xcecli_ctx_t, xce_taskrun_t*);
|
||||
extern int xcecli_notify_taskstart(xcecli_ctx_t, xce_task_pidinfo_t*);
|
||||
extern int xcecli_notify_taskfail(xcecli_ctx_t, xce_task_failinfo_t*);
|
||||
|
|
|
|||
|
|
@ -185,6 +185,19 @@ extern int xce_upgrade_config_req_extract_raw(const uint8_t *pkt_data,
|
|||
size_t *out_len,
|
||||
uint8_t **out_sha256);
|
||||
|
||||
/*
|
||||
* xce_upgradeconfig_req_raw: send OP_upgradeConfig carrying raw byte arrays
|
||||
* (sha256 + the serialised ClusterConfig blob). Bypasses the generated
|
||||
* encode_UpgradeConfigRequest path (a no-op for the sha256/bytes Data fields
|
||||
* — same class of limitation as xce_replconfigpush_req_raw) by constructing
|
||||
* the capnp message directly via capn_new_list8 + capn_setv8 + write_Request.
|
||||
* Ordinary CLIENT request: seq/from are populated the same way xce_request()
|
||||
* populates them for every other client op.
|
||||
*/
|
||||
extern int xce_upgradeconfig_req_raw(endpoint_t *ep,
|
||||
const uint8_t *sha256, size_t sha256_len,
|
||||
const uint8_t *bytes, size_t bytes_len);
|
||||
|
||||
/* ── Client-Side Transparent Retry tunables (Req 6.1, 6.2) ──────────────────
|
||||
*
|
||||
* xce_retry_config_t: configurable retry parameters with safe production
|
||||
|
|
|
|||
|
|
@ -28,6 +28,17 @@ extern int xce_update_resvals(xce_ctx_t*, int, xce_resval_t**);
|
|||
extern int xce_jobctrl(xce_ctx_t*, xce_jobctrl_t*);
|
||||
extern int xce_queuectrl(xce_ctx_t*, xce_queuectrl_t*);
|
||||
extern int xce_hostctrl(xce_ctx_t*, xce_hostctrl_t*);
|
||||
/* xce_upgradeconfig: send OP_upgradeConfig (sha256 + serialised ClusterConfig
|
||||
* blob) via xce_upgradeconfig_req_raw and wait for the reply. Returns the
|
||||
* server's reply status (0 on success; a negative errno/E_XCE_* code on
|
||||
* failure — the server logs the human-readable reason, see
|
||||
* do_upgrade_config_with_bytes/send_upgrade_error, config_repl.c). Mirrors
|
||||
* xce_hostctrl/xce_queuectrl's shape; the richer UpgradeConfigReply fields
|
||||
* (version_tag/acked_count/error text) are not surfaced to the caller — no
|
||||
* consumer of this API currently needs them beyond success/failure. */
|
||||
extern int xce_upgradeconfig(xce_ctx_t*, const uint8_t *sha256,
|
||||
size_t sha256_len, const uint8_t *bytes,
|
||||
size_t bytes_len);
|
||||
extern int xce_runtask(xce_ctx_t*, xce_taskrun_t*);
|
||||
extern int xce_update_jusage(xce_ctx_t*, uint64_t, int, int*, int, int*, int,
|
||||
xce_rusage_t**);
|
||||
|
|
|
|||
|
|
@ -778,6 +778,25 @@ int xcecli_hostctrl(xcecli_ctx_t c, xce_hostctrl_t* ctrlinfo)
|
|||
return rc;
|
||||
}
|
||||
|
||||
int xcecli_upgradeconfig(xcecli_ctx_t c, const uint8_t *sha256,
|
||||
size_t sha256_len, const uint8_t *bytes,
|
||||
size_t bytes_len)
|
||||
{
|
||||
api_ctx_t* ctx = (api_ctx_t*)c;
|
||||
int rc;
|
||||
|
||||
if ((ctx == NULL) || (sha256 == NULL) || (sha256_len == 0)
|
||||
|| (bytes == NULL) || (bytes_len == 0)) {
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
rc = XCECLI_FAILOVER_CALL(ctx, /*is_write=*/true,
|
||||
xce_upgradeconfig(&(ctx->ctx), sha256,
|
||||
sha256_len, bytes,
|
||||
bytes_len));
|
||||
return rc;
|
||||
}
|
||||
|
||||
int xcecli_runtask(xcecli_ctx_t c, xce_taskrun_t* info)
|
||||
{
|
||||
api_ctx_t* ctx = (api_ctx_t*)c;
|
||||
|
|
|
|||
|
|
@ -499,6 +499,94 @@ int xce_replconfigpush_req_raw(endpoint_t *ep, xce_peerinfo_t *from,
|
|||
return (enq_rc < 0) ? enq_rc : 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* xce_upgradeconfig_req_raw: send OP_upgradeConfig with raw byte arrays.
|
||||
*
|
||||
* Bypasses encode_UpgradeConfigRequest (a literal no-op for the sha256/bytes
|
||||
* Data fields — see the doc comment above upgrade_config_req_s in typemap.h;
|
||||
* confirmed by inspection of the generated xce.build/lib/controller.capnp.c).
|
||||
* Instead, constructs the struct Request / struct UpgradeConfigRequest
|
||||
* directly using the capnp wire-format API (capn_new_list8 + capn_setv8),
|
||||
* mirroring xce_replconfigpush_req_raw's exact bypass pattern.
|
||||
*
|
||||
* Unlike that controller-to-controller helper, this is an ordinary CLIENT
|
||||
* request: seq and from are populated the same way the generic xce_request()
|
||||
* populates them for every other client op (endpoint_getseq(ep), ep->peer),
|
||||
* not passed in by the caller.
|
||||
*/
|
||||
int xce_upgradeconfig_req_raw(endpoint_t *ep,
|
||||
const uint8_t *sha256, size_t sha256_len,
|
||||
const uint8_t *bytes, size_t bytes_len)
|
||||
{
|
||||
struct capn c;
|
||||
struct capn_segment *cs;
|
||||
struct Request rq;
|
||||
struct UpgradeConfigRequest up_req;
|
||||
UpgradeConfigRequest_ptr up_ptr;
|
||||
Request_ptr rp;
|
||||
io_packet_t *pkt;
|
||||
int rc;
|
||||
|
||||
if (ep == NULL)
|
||||
return -EINVAL;
|
||||
|
||||
if (sha256 == NULL || sha256_len == 0 || bytes == NULL || bytes_len == 0)
|
||||
return -EINVAL;
|
||||
|
||||
capn_init_malloc(&c);
|
||||
cs = capn_root(&c).seg;
|
||||
|
||||
memset(&rq, 0, sizeof(rq));
|
||||
memset(&up_req, 0, sizeof(up_req));
|
||||
|
||||
/* sha256: write as a List(UInt8) — capn_list8 and capn_data share {capn_ptr p} */
|
||||
{
|
||||
capn_list8 sha_list = capn_new_list8(cs, (int)sha256_len);
|
||||
capn_setv8(sha_list, 0, sha256, (int)sha256_len);
|
||||
up_req.sha256.p = sha_list.p;
|
||||
}
|
||||
|
||||
/* config bytes */
|
||||
{
|
||||
capn_list8 bytes_list = capn_new_list8(cs, (int)bytes_len);
|
||||
capn_setv8(bytes_list, 0, bytes, (int)bytes_len);
|
||||
up_req.bytes.p = bytes_list.p;
|
||||
}
|
||||
|
||||
up_ptr = new_UpgradeConfigRequest(cs);
|
||||
write_UpgradeConfigRequest(&up_req, up_ptr);
|
||||
|
||||
rq.opcode = OPCode_upgradeConfig;
|
||||
rq.seq = endpoint_getseq(ep);
|
||||
rq.body_which = Request_body_upgradeconfigrequestmsg;
|
||||
rq.body.upgradeconfigrequestmsg = up_ptr;
|
||||
encode_PeerInfo_ptr(cs, &rq.from, ep->peer);
|
||||
|
||||
rp = new_Request(cs);
|
||||
write_Request(&rq, rp);
|
||||
capn_setp(capn_root(&c), 0, rp.p);
|
||||
|
||||
pkt = (io_packet_t *)calloc(1, sizeof(*pkt));
|
||||
if (!pkt) {
|
||||
capn_free(&c);
|
||||
return -ENOMEM;
|
||||
}
|
||||
|
||||
pkt->sz = capn_size(&c);
|
||||
rc = alloc_packet(pkt);
|
||||
if (rc < 0) {
|
||||
free(pkt);
|
||||
capn_free(&c);
|
||||
return -errno;
|
||||
}
|
||||
|
||||
capn_write_mem(&c, (uint8_t *)pkt->data, pkt->sz, 0);
|
||||
capn_free(&c);
|
||||
|
||||
int enq_rc = endpoint_enq(ep, &(pkt->link));
|
||||
return (enq_rc < 0) ? enq_rc : 0;
|
||||
}
|
||||
|
||||
/* wal_entry_to_wire: fill a wire WalCatchupEntry from a C entry's header
|
||||
* fields and raw payload bytes. Sub-entry nesting is one level deep —
|
||||
* elements written here never carry their own entries list. */
|
||||
|
|
|
|||
|
|
@ -647,6 +647,36 @@ int xce_hostctrl(xce_ctx_t* ctx, xce_hostctrl_t* ctrl)
|
|||
return status;
|
||||
}
|
||||
|
||||
int xce_upgradeconfig(xce_ctx_t* ctx, const uint8_t *sha256,
|
||||
size_t sha256_len, const uint8_t *bytes,
|
||||
size_t bytes_len)
|
||||
{
|
||||
int rc;
|
||||
int status;
|
||||
|
||||
if (ctx == NULL) {
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
if (ctx->state != XCE_CTX_RUN) {
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
rc = xce_upgradeconfig_req_raw(ctx->ep, sha256, sha256_len, bytes,
|
||||
bytes_len);
|
||||
if (rc < 0) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
struct timespec _dl; cbuf_deadline_10s(&_dl);
|
||||
rc = cbuf_timedget(&(ctx->wbuf), &status, &_dl);
|
||||
if (rc < 0) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
return status;
|
||||
}
|
||||
|
||||
int xce_runtask(xce_ctx_t* ctx, xce_taskrun_t* runinfo)
|
||||
{
|
||||
int rc;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue