From 959b0cdacb054a3cc0fe8557ba39cac6fb8c52d7 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Sat, 25 Jul 2026 15:48:20 +0000 Subject: [PATCH] feat(config-upgrade-snapshot): live config-upgrade soak infra + fix bug-1468/bug-1469 (task 8.6) Build the live 3-node chaotic-multi-kill config-upgrade soak scenario (run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1, 6.4. This needed genuinely new client-side infrastructure: a hand-rolled capnp send path for OP_upgradeConfig (the auto-generated encoder is a no-op for its Data fields, worked around the same way xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig CLI commands. Building and running this live surfaced two real production bugs, both fixed here: - bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's primary/standby dispatch guard in cluster.c, making config upgrades unreachable from any network client. - bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called config_activate_committed(), so the node that originates a config upgrade never refreshed its own in-memory state as part of committing it. Killing that node before some later, unrelated event happened to refresh it left its checkpoint permanently stale -- the first live, empirical reproduction of bug-857's originally hypothesized failure mode. Fixed by activating before the marker-triggered checkpoint write, since checkpoint content is built from live runtime tables that only reflect the reload after activation runs. A separate, narrower checkpoint divergence (bug-1471, not config-related) remains open and still blocks a clean soak PASS -- task 8.6 stays in-progress until it's resolved. --- .claude/skills/xce-test-ha/SKILL.md | 81 ++++- .claude/skills/xce-test-ha/run-ha-soak.sh | 320 +++++++++++++++++++ .kiro/specs/config-upgrade-snapshot/tasks.md | 6 + src/controller/cluster.c | 28 +- src/controller/persist.c | 110 +++++++ src/controller/testsuite/test.c | 149 ++++++++- src/controller/testsuite/test_config_repl.c | 169 +++++++++- src/include/client.h | 8 + src/include/request.h | 13 + src/include/xceapi.h | 11 + src/lib/client.c | 19 ++ src/lib/request.c | 88 +++++ src/lib/xceapi.c | 30 ++ 13 files changed, 1022 insertions(+), 10 deletions(-) diff --git a/.claude/skills/xce-test-ha/SKILL.md b/.claude/skills/xce-test-ha/SKILL.md index ae3897c..53beb07 100644 --- a/.claude/skills/xce-test-ha/SKILL.md +++ b/.claude/skills/xce-test-ha/SKILL.md @@ -266,6 +266,83 @@ invariant is asserted with an immediate `fail()` inside `run_array_chaos_retry_round()`, so a violation aborts the run before this line is ever reached. +### `--config-upgrade` mode: live configuration upgrade under chaotic failover (config-upgrade-snapshot task 8.6) + +Drives a real `OP_upgradeConfig` RPC (the new `upgradeconfig` `xce_test` +client command, backed by `xcecli_upgradeconfig` → +`xce_upgradeconfig_req_raw`) against the live primary, submitted in the +**background** and interleaved with a bounded chaotic kill/restart sequence +(`XCE_HA_CONFIGUPGRADE_KILLS` rounds, default 3) — relying on the client's own +transparent failover (`XCECLI_FAILOVER_CALL`) to survive a primary change +mid-call, exactly like `--array-chaos-retry` relies on it for a job-array +submit. See `run_config_upgrade_round()`'s header comment (`run-ha-soak.sh`) +for the full mechanism trace. + +Sequence: + +1. Joins a synthetic worker host (wire-level `join`, same pattern + `--factstream` uses — no real `xcecd` process) and admin-closes it via the + new `hostctrl -c` command, giving Req 6.4 a concrete admin bit to prove + survives the upgrade. +2. Submits an initial batch of jobs — committed state the upgrade must + preserve. +3. Builds a second, "upgrade-target" `cluster.bin` from a scratch copy of the + **whole** `src/configs/` tree (not just `queues.yaml` — `xceadmin config + build` needs the full directory) with one deliberate, observable change: + the `night` queue's description text. Deliberately an **update to an + existing queue**, not an added one — see `run_config_upgrade_round()`'s + own comment and `.wolf/buglog.json` bug-1469: a live (non-restart) config + upgrade has no working mechanism today to create a *new* runtime queue + object in a member's `c->qmgr` (`build_queues()`, queue.c, only ever runs + once at process boot); that is a genuine, deep gap in tasks 1-7's + materialization logic, confirmed live and flagged for their owners, but + out of this task's scope to fix. An update to an existing queue's + description exercises exactly what Req 4.1-4.3/6.1/6.4 ask this scenario + to prove without depending on that separate, unrelated mechanism. +4. Submits the upgrade in the background, then runs the chaotic kill/restart + sequence — restarting the previously-downed node WITHOUT waiting for full + reconvergence before the next kill (like `--chaotic`), so the upgrade + genuinely has a chance to interleave with an in-progress recovery, not just + complete cleanly before or after every kill. +5. After the cluster reconverges 3-of-3 and the background submission + confirms success (retried once against the settled cluster if it did not + report success while kills were still landing), hard-asserts (`fail()`, + not count-and-tolerate — a definite correctness claim, like + `--force-snapshot`): + - **materialized-state identity** across all 3 members — byte-diff + (`snapdiff`) of each member's latest on-disk checkpoint file; + - **active-configuration identity** — `configcompare` (task 8.4's tool, + its first live use here) across all 3 members' `conf/current`; + - the synthetic host's **admin-CLOSED status specifically survives**, + checked DIRECTLY per member (not merely inferred from the two identity + checks above) by rotating primary through all 3 nodes in turn + (`force_elect_node`) and asserting `workerlist` shows it CLOSED while + each one serves — proving a standby-turned-primary materializes the + admin bit correctly too, not just whichever node happened to be primary + when the upgrade committed. + +Single dedicated round, not a loop — like `--force-snapshot`/`--array-soak`. + +```bash +SKILL_DIR=.claude/skills/xce-test-ha +nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --config-upgrade ~/tmp/xce-ha-configupgrade +# tune kill-round count / synthetic host name +XCE_HA_CONFIGUPGRADE_KILLS=5 XCE_HA_CONFIGUPGRADE_HOST=my-host \ + nix develop ./src --command bash "$SKILL_DIR/run-ha-soak.sh" --config-upgrade ~/tmp/xce-ha-configupgrade +``` + +Verdict line: + +``` +[xce-test-ha] PASS (config-upgrade): 3 jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members +``` + +Like `--array-soak`/`--array-chaos-retry`, `result` is unconditionally +`PASS` here — every hard invariant is asserted with an immediate `fail()` +inside `run_config_upgrade_round()`, so a violation aborts the run before +this line is ever reached; there is no `verdict_emit`/bug-924 exclusion +tally for this mode. + ### Verdict self-test (no cluster) The verdict parser has a deterministic, pure-bash self-test that feeds fixture @@ -361,7 +438,7 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK | Env var | Meaning | Default | |---|---|---| | `XCE_HA_TOP` | deploy directory | `/home/claude/tmp/xce-ha-deploy` | -| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, or `array-chaos-retry` (its `--flag` wins) | `sequential` | +| `XCE_HA_MODE` | `sequential`, `chaotic`, `force-snapshot`, `factstream`, `array-soak`, `array-chaos-retry`, or `config-upgrade` (its `--flag` wins) | `sequential` | | `XCE_HA_BIN` | `xce_controller` binary (point at real-fs release for the gate) | `/src/xce.build/controller/xce_controller` | | `XCE_HA_XT` | `xce_test` binary | `/src/xce.build/controller/xce_test` | | `XCE_HA_ROUNDS` | kill/restart rounds (5–50) | `6` | @@ -381,6 +458,8 @@ XCE_HA_ROUNDS=10 XCE_HA_TOP=/tmp/my-ha-run nix develop ./src --command bash "$SK | `XCE_HA_ARRAYRETRY_ROUNDS` | `--array-chaos-retry` kill-during-submit rounds | `5` | | `XCE_HA_ARRAYRETRY_SIZE` | `--array-chaos-retry` element count of the array each round submits | `6` | | `XCE_HA_ARRAYRETRY_KILL_DELAY` | `--array-chaos-retry` seconds slept after backgrounding the submit before killing the primary | `0` | +| `XCE_HA_CONFIGUPGRADE_KILLS` | `--config-upgrade` kill/restart rounds interleaved with the upgrade submission | `3` | +| `XCE_HA_CONFIGUPGRADE_HOST` | `--config-upgrade` name of the synthetic worker host joined and admin-closed before the upgrade | `cfgupgrade-host` | ## What the driver does (maps to the requested steps) diff --git a/.claude/skills/xce-test-ha/run-ha-soak.sh b/.claude/skills/xce-test-ha/run-ha-soak.sh index 6199258..3d42830 100755 --- a/.claude/skills/xce-test-ha/run-ha-soak.sh +++ b/.claude/skills/xce-test-ha/run-ha-soak.sh @@ -32,6 +32,25 @@ # own log that a snapshot install actually happened (not # just a claim), then asserts zero walcompare divergence # and zero job loss. Single dedicated round, not a loop. +# --config-upgrade config-upgrade-snapshot task 8.6 (Req 4.1, 4.3, 6.1, +# 6.4): admin-closes a synthetic worker host, then submits a +# live configuration upgrade (OP_upgradeConfig, via the new +# `upgradeconfig` xce_test command) CONCURRENTLY with a +# bounded chaotic kill/restart sequence (XCE_HA_CONFIGUPGRADE_ +# KILLS rounds), relying on the client's own transparent +# failover (XCECLI_FAILOVER_CALL) to survive a primary +# change mid-call. After the cluster reconverges, hard- +# asserts (fail(), not count-and-tolerate — like +# --force-snapshot, this is a definite correctness claim): +# (1) materialized-state identity across all 3 members via +# byte-diff of each member's latest on-disk checkpoint file +# (snapdiff); (2) active-configuration identity via +# configcompare (task 8.4's tool, its first live use); (3) +# the admin-closed host's status specifically survives, +# checked DIRECTLY per member (not inferred from (1)/(2)) +# by rotating primary through all 3 nodes and querying +# workerlist against each. Single dedicated round, not a +# loop. # --factstream raft-apply-decoupling task 7.2 (Req 10.2/10.4): drives a # continuous WORKER-FACT stream (jstart/jfinish/jfail — real # wire-protocol worker reports, distinct from job-submission @@ -154,6 +173,14 @@ # (default 0 — immediate, to maximize "chaotic" # uncertainty about whether the WAL append/ # commit happened before the kill) +# XCE_HA_CONFIGUPGRADE_KILLS kill/restart rounds interleaved with the +# config-upgrade submission (default 3, +# independent of the general --chaotic ROUNDS +# knob — this is a single dedicated round, not +# a soak loop) +# XCE_HA_CONFIGUPGRADE_HOST name of the synthetic worker host joined and +# admin-closed before the upgrade, used to prove +# Req 6.4 (default "cfgupgrade-host") set -u # ── Mode / flag parsing (flags may precede the positional [ROUNDS] [DEPLOY_DIR]) ─ @@ -174,6 +201,7 @@ while [ $# -gt 0 ]; do --factstream) MODE="factstream" ;; --array-soak) MODE="array-soak" ;; --array-chaos-retry) MODE="array-chaos-retry" ;; + --config-upgrade) MODE="config-upgrade" ;; --selftest) exec bash "$SELF_DIR/soak-verdict-selftest.sh" ;; *[!0-9]*|'') ARGS+=("$1") ;; # non-integer ⇒ deploy dir *) ROUNDS_CLI="$1" ;; # bare integer ⇒ rounds @@ -1016,6 +1044,276 @@ run_force_snapshot_round() { walcheck_count "${#ALL_JOBIDS[@]}" } +# ── Config-upgrade primitives (config-upgrade-snapshot task 8.6) ──────────── + +# Locate the xceadmin binary — same search list deploy-ha.sh uses (meson +# build output under whichever profile built it, or a nix-installed prefix). +# Needed here because this mode builds a SECOND cluster.bin itself (the +# upgrade target), unlike every other mode which only ever consumes the one +# deploy-ha.sh already built into ctrl-1/conf during Step 2. +find_xceadmin() { + local candidate + for candidate in \ + "$REPO/src/xce.build/xceadmin/.venv/bin/xceadmin" \ + "$REPO/src/build.release/xceadmin/.venv/bin/xceadmin" \ + "$REPO/src/build.dbg/xceadmin/.venv/bin/xceadmin" \ + "$REPO/src/result/bin/xceadmin" \ + "$(command -v xceadmin 2>/dev/null || true)" + do + if [ -n "$candidate" ] && [ -x "$candidate" ]; then + echo "$candidate" + return 0 + fi + done + return 1 +} + +# Echo a member's latest (highest-LSN) on-disk checkpoint file +# ($TOP/controller//data/snap/snap-.capnp — persist.c's +# p->snap_path, "%s/data/snap" under the node's workdir/name), or fail with +# rc!=0 if none exists yet. Filenames are fixed-width %016 hex, so +# lexicographic sort == LSN-numeric sort. +latest_snapshot_file() { + local node="$1" f + f="$(ls -1 "$TOP/controller/$node/data/snap"/snap-*.capnp 2>/dev/null | sort | tail -1)" + [ -n "$f" ] || return 1 + echo "$f" +} + +# config-upgrade-snapshot task 8.6 (Req 4.1, 4.3, 6.1, 6.4): drive a live +# configuration upgrade (OP_upgradeConfig, the new client built in this same +# task — lib/request.c xce_upgradeconfig_req_raw / lib/xceapi.c +# xce_upgradeconfig / lib/client.c xcecli_upgradeconfig, testsuite/test.c +# cmd_upgradeconfig) while chaotically killing/restarting the primary, then +# hard-assert (fail(), not count-and-tolerate — mirroring +# run_force_snapshot_round's own convention: this is a definite correctness +# claim, not a racy/sometimes-expected outcome class): +# +# - materialized-state identity across all 3 members (Req 4.1) — via +# byte-diff (snapdiff) of each member's latest on-disk checkpoint file, +# the SAME tooling design.md's Requirements Traceability row 6.1 cites +# (snapdiff_paths, configcompare) — NOT live per-node joblist/workerlist +# RPCs, because this system's read model is primary-only: OPCode_get +# (which joblist/workerlist/reslist all use) is absent from +# ha_dispatch_gate's CTRL_STANDBY allow-list (ha.c), so a standby cannot +# answer it directly — confirmed by direct inspection, not assumed. +# - active-configuration identity across all 3 members (Req 4.2/4.3) — via +# configcompare (task 8.4's tool), its first live use. +# - a host's administratively-set status specifically survives (Req 6.4), +# checked DIRECTLY per member — not merely inferred from the aggregate +# identity checks above — by rotating primary through each of the 3 +# nodes in turn (force_elect_node, already used for this script's +# leadership-coverage step) and querying workerlist against whichever +# node is currently serving, asserting the host shows CLOSED each time. +# +# One dedicated round, not a loop — like run_force_snapshot_round. +run_config_upgrade_round() { + local kills="${XCE_HA_CONFIGUPGRADE_KILLS:-3}" + local host="${XCE_HA_CONFIGUPGRADE_HOST:-cfgupgrade-host}" + local xceadmin scratch_dir scratch_bin + + xceadmin="$(find_xceadmin)" \ + || fail "config-upgrade: xceadmin binary not found (run /xce-build all or /xce-build xceadmin)" + + # 1. Join + admin-close a synthetic worker host BEFORE the upgrade, so + # Req 6.4 has a concrete admin bit to prove survives. Wire-level + # synthetic worker join — the same pattern run_factstream_round + # already uses (`set myname ` + `worker` + `join`); no real + # xcecd process is ever started. do_hostcontrol_op (worker.c) requires + # the host to already exist in the live worker table (-ENOENT + # otherwise) — this join step exists precisely to satisfy that, since + # hosts.yaml config entries do NOT auto-populate workermgr (a worker_t + # is created only by a live join, requirements.md's own Boundary + # Context note). + log "config-upgrade: joining synthetic host $host" + local jout + jout="$(printf 'set masters %s\nset myname %s\nworker\njoin\nquit\n' "$MASTERS" "$host" | xtest)" + echo "$jout" | grep -q "host join ok" \ + || fail "config-upgrade: synthetic host $host failed to join: $jout" + + log "config-upgrade: admin-closing $host via hostctrl" + local hout + hout="$(printf 'set masters %s\nset myname soak\nclient\nwait-primary %s\nhostctrl -c -h %s\nquit\n' \ + "$MASTERS" "$WAITMS" "$host" | xtest)" + echo "$hout" | grep -q "Operation on host $host done" \ + || fail "config-upgrade: hostctrl -c on $host failed: $hout" + log "config-upgrade: $host admin-closed" + + # 2. Initial batch of jobs — committed state the upgrade must preserve. + submit_jobs "$NJOBS" || fail "config-upgrade: initial submit failed" + verify_no_loss || fail "config-upgrade: job loss before the upgrade started" + + # 3. Build the upgrade-target config: a scratch COPY of the whole + # src/configs/ tree (schema/ subdir included, exactly mirroring what + # deploy-ha.sh itself passes to `xceadmin config build`) with one + # deliberate, observable change — the "night" queue's description + # text — so the post-upgrade identity check is a real assertion, not + # a tautology. + # + # Deliberately an UPDATE to an EXISTING queue, not an ADDED queue + # (bug-1469 finding, .wolf/buglog.json): a live (non-restart) config + # upgrade has no working mechanism to create a NEW runtime queue_t + # object in any member's c->qmgr — build_queues() (queue.c), the only + # code that ever populates c->qmgr from parsed config, runs exactly + # once at process boot (xce_config_init) and is unconditional/blind- + # create, unsafe to re-invoke live. That is a genuine, deep, + # confirmed gap in tasks 1-7's materialization logic (requires new, + # carefully-designed production code plus a queue-removal-semantics + # design decision this task is not chartered to make) — reproduced + # live TWICE (before and after config-upgrade-snapshot task 6.2's own + # activation-ordering fix), fully root-caused, and flagged for + # tasks 1-7's owners, but explicitly OUT OF SCOPE for task 8.6 to fix. + # An UPDATE to an existing queue's description exercises exactly what + # Requirements 4.1-4.3/6.1/6.4 ask this task to prove (materialized + # job/host state + active-configuration identity across members, + # surviving chaotic failover) without depending on that separate, + # unrelated, unfixed mechanism — collect_queue_frames' QueueAdminState + # frame only ever carries name+admin-status (persist.c:1895), never + # description/priority/etc., so this change is invisible to the + # queue-entity section either way and is carried purely by the + # checkpoint's embedded raw-config-bytes section (task 2.3, + # read_active_config_bytes — always a fresh disk re-read, unaffected + # by the qmgr-sync gap) and by configcompare's config-file-hash + # comparison — both of which are exactly the "active configuration + # identical" mechanisms Requirement 4.2/4.3 actually asks about. + scratch_dir="$TOP/.cfgupgrade-src" + scratch_bin="$TOP/.cfgupgrade-src.bin" + rm -rf "$scratch_dir" "$scratch_bin" + cp -r "$REPO/src/configs" "$scratch_dir" + sed -i 's/description: "night queue"/description: "night queue (config-upgrade-snapshot task 8.6 soak marker)"/' \ + "$scratch_dir/queues.yaml" + grep -q "task 8.6 soak marker" "$scratch_dir/queues.yaml" \ + || fail "config-upgrade: sed did not find the expected 'night queue' description line to mark — src/configs/queues.yaml may have changed shape" + "$xceadmin" config build "$scratch_dir" -o "$scratch_bin" \ + || fail "config-upgrade: xceadmin config build failed for the upgrade-target config" + log "config-upgrade: built upgrade-target cluster.bin with an updated 'night' queue description: $scratch_bin" + + # 4. Submit the upgrade CONCURRENTLY with a bounded kill/restart + # sequence — at least one kill/restart genuinely overlaps the upgrade + # RPC. xcecli_upgradeconfig's XCECLI_FAILOVER_CALL wrapper (Part A) is + # relied on to survive a primary change mid-call; xtest's own 60s + # timeout bounds the background job regardless. + local upgrade_log="$TOP/logs/config-upgrade-submit.out" + : >"$upgrade_log" + ( printf 'set masters %s\nset myname cfgupgrade\nclient\nwait-primary %s\nupgradeconfig %s\nquit\n' \ + "$MASTERS" "$WAITMS" "$scratch_bin" | xtest ) >"$upgrade_log" 2>&1 & + local upgrade_pid=$! + log "config-upgrade: submitted upgradeconfig in the background (pid=$upgrade_pid, log=$upgrade_log)" + + local DOWN="" round PRIMARY PREV_TERM NEWP PID + for round in $(seq 1 "$kills"); do + log "config-upgrade: kill round $round/$kills" + if [ -n "$DOWN" ]; then + log "config-upgrade: restarting previously-downed $DOWN (no convergence wait — overlapping, like chaotic mode, so the upgrade genuinely interleaves with an in-progress recovery)" + start_node "$DOWN" + fi + PRIMARY="$(primary_from_log)" + if [ -z "$PRIMARY" ]; then + wait_for_primary_log || log "config-upgrade: round $round: no primary elected yet" + PRIMARY="$(primary_from_log)" + fi + if [ -z "$PRIMARY" ]; then + log "config-upgrade: round $round: no primary to kill — skipping" + continue + fi + record_primary "$PRIMARY" + PREV_TERM="$(latest_primary_term)"; PREV_TERM="${PREV_TERM:-0}" + PID="$(pgrep -f -- "$(node_pattern "$PRIMARY")")" + if [ -n "$PID" ]; then + log "config-upgrade: round $round: killing primary $PRIMARY (term=$PREV_TERM)" + kill -9 $PID 2>/dev/null + DOWN="$PRIMARY" + else + log "config-upgrade: round $round: primary $PRIMARY already gone" + fi + sleep 1 + NEWP="$(wait_for_new_primary "$PREV_TERM")" + if [ -n "$NEWP" ]; then + log "config-upgrade: round $round: new primary elected: $NEWP (term=$(latest_primary_term))" + record_primary "$NEWP" + else + log "config-upgrade: round $round: no new primary within window (possible election stall)" + fi + done + + # Restore full 3-of-3 before judging anything. + if [ -n "$DOWN" ]; then + log "config-upgrade: restarting last-downed $DOWN and settling" + start_node "$DOWN" + fi + wait_for_full_convergence \ + || fail "config-upgrade: cluster did not reconverge to 3-of-3 after the chaotic kill sequence" + + log "config-upgrade: waiting for the background upgrade submission to finish" + local waited=0 + while kill -0 "$upgrade_pid" 2>/dev/null && [ "$waited" -lt 65 ]; do + sleep 1 + waited=$((waited + 1)) + done + wait "$upgrade_pid" 2>/dev/null + if ! grep -q "Config upgrade submitted" "$upgrade_log"; then + log "config-upgrade: submission did not report success while interleaved with the kills — retrying once against the now-settled, reconverged cluster (prior log: $(cat "$upgrade_log"))" + printf 'set masters %s\nset myname cfgupgrade\nclient\nwait-primary %s\nupgradeconfig %s\nquit\n' \ + "$MASTERS" "$WAITMS" "$scratch_bin" | xtest >"$upgrade_log" 2>&1 + grep -q "Config upgrade submitted" "$upgrade_log" \ + || fail "config-upgrade: upgradeconfig never succeeded, even after the chaotic sequence settled: $(cat "$upgrade_log")" + fi + log "config-upgrade: upgrade submission confirmed committed" + + # Give every member a moment to apply the marker and materialize its own + # upgrade snapshot (Req 1.1/1.5 — synchronous FROM EACH MEMBER'S OWN + # apply-pump perspective, but a behind/just-restarted member's own apply + # can still lag the marker's commit by a few seconds). + sleep 5 + + # 5. Materialized-state identity (Req 4.1) + job survival. + verify_no_loss \ + || fail "config-upgrade: a committed job was LOST across the upgrade+chaotic-kill scenario" + + local f1 f2 f3 + f1="$(latest_snapshot_file ctrl-1)" || fail "config-upgrade: ctrl-1 has no on-disk checkpoint after the upgrade" + f2="$(latest_snapshot_file ctrl-2)" || fail "config-upgrade: ctrl-2 has no on-disk checkpoint after the upgrade" + f3="$(latest_snapshot_file ctrl-3)" || fail "config-upgrade: ctrl-3 has no on-disk checkpoint after the upgrade" + log "config-upgrade: comparing latest checkpoint files: $f1 vs $f2 vs $f3" + local sdout + sdout="$(printf 'snapdiff %s %s\nquit\n' "$f1" "$f2" | xtest)" + echo "$sdout" | grep -q "IDENTICAL" \ + || fail "config-upgrade: materialized-state DIVERGENCE ctrl-1 vs ctrl-2: $sdout" + sdout="$(printf 'snapdiff %s %s\nquit\n' "$f1" "$f3" | xtest)" + echo "$sdout" | grep -q "IDENTICAL" \ + || fail "config-upgrade: materialized-state DIVERGENCE ctrl-1 vs ctrl-3: $sdout" + log "config-upgrade: materialized state IDENTICAL across all 3 members (checkpoint byte-identity)" + + # 6. Configuration identity (Req 4.2/4.3). configcompare_paths takes each + # member's workdir/name conf-root DIRECTORY — $TOP/controller/ctrl-N, + # NOT .../ctrl-N/conf — it appends "/conf/current" itself + # (configcompare.c's read_member_config_sha256; verified by reading + # the tool's source directly before this invocation). + local ccout + ccout="$(printf 'configcompare %s %s %s\nquit\n' \ + "$TOP/controller/ctrl-1" "$TOP/controller/ctrl-2" "$TOP/controller/ctrl-3" | xtest)" + echo "$ccout" | grep -q "IDENTICAL across 3 members" \ + || fail "config-upgrade: active-CONFIGURATION divergence across members: $ccout" + log "config-upgrade: active configuration IDENTICAL across all 3 members" + + # 7. Host admin-status survival (Req 6.4) — checked DIRECTLY per member, + # not inferred from the aggregate identity checks above. Rotate + # primary through each of the 3 nodes in turn (force_elect_node + # no-ops cheaply if the target is already primary) and assert $host + # shows CLOSED via workerlist each time — the only way to query a + # SPECIFIC member's own live view directly, since a standby cannot + # answer OPCode_get (workerlist) itself. + local n wl + for n in "${NODES[@]}"; do + force_elect_node "$n" || fail "config-upgrade: could not force-elect $n to directly verify its own host admin-status view" + wl="$(printf 'set masters %s\nset myname soak\nclient\nwait-primary %s\nworkerlist\nquit\n' \ + "$MASTERS" "$WAITMS" | xtest)" + echo "$wl" | grep -qE "^${host}[[:space:]]+CLOSED" \ + || fail "config-upgrade: host $host is NOT admin-CLOSED while $n served as primary: $wl" + log "config-upgrade: $host confirmed CLOSED while $n served as primary" + done +} + # ── Fact-stream primitives (raft-apply-decoupling task 7.2, Req 10.2/10.4) ─── # # Drives a continuous stream of WORKER-FACT messages (jstart/jfinish/jfail — @@ -1925,6 +2223,28 @@ if [ "$MODE" = "array-chaos-retry" ]; then exit 0 fi +# ── Config-upgrade mode: config-upgrade-snapshot task 8.6 ──────────────────── +# Opt-in (--config-upgrade / XCE_HA_MODE=config-upgrade). See +# run_config_upgrade_round's own header comment for the full mechanism trace. +# Single dedicated round, not a loop — like --array-soak/--array-chaos-retry +# (not --force-snapshot's verdict_emit-based counted-verdict style), every +# hard invariant (job survival, materialized-state identity via snapdiff, +# active-configuration identity via configcompare, host admin-status +# survival) is asserted with an immediate fail() inside +# run_config_upgrade_round() itself, so reaching the PASS line below already +# means every check passed — no scan_bug924/V_BUG924 tally here, matching +# array-soak/array-chaos-retry's convention (not force-snapshot/factstream's, +# which feed V_BUG924 into a verdict_emit exclusion). Does NOT run the +# sequential loop or leadership-coverage maneuver. +if [ "$MODE" = "config-upgrade" ]; then + log "MODE=config-upgrade — driving a configuration upgrade interleaved with chaotic multi-kill failover" + run_config_upgrade_round + log "PASS (config-upgrade): ${#ALL_JOBIDS[@]} jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members" + cleanup + trap - EXIT INT TERM + exit 0 +fi + # ── Chaotic mode: overlapping-kill rounds → isolated verdict ────────────────── # Opt-in (--chaotic / XCE_HA_MODE=chaotic). Kills land inside election/catch-up # windows; outcomes are COUNTED (committed divergence + committed job loss) with diff --git a/.kiro/specs/config-upgrade-snapshot/tasks.md b/.kiro/specs/config-upgrade-snapshot/tasks.md index 61c393a..a9256d5 100644 --- a/.kiro/specs/config-upgrade-snapshot/tasks.md +++ b/.kiro/specs/config-upgrade-snapshot/tasks.md @@ -149,6 +149,12 @@ ## Implementation Notes +- **Task 8.6 progress (infrastructure + two real production bugs found and fixed; task NOT YET complete — one more bug blocks a clean live-soak PASS):** building this task's live 3-node chaotic-upgrade soak required substantial new infrastructure that never existed anywhere in this codebase: a client-side send path for `OP_upgradeConfig` (`xce_upgradeconfig_req_raw`/`xce_upgradeconfig`/`xcecli_upgradeconfig` — the auto-generated capnp encoder for this request type is a literal no-op for its `Data` fields, worked around via a hand-rolled bypass mirroring the existing `xce_replconfigpush_req_raw` idiom), a `cmd_hostctrl`/`cmd_upgradeconfig` CLI pair in `test.c`, and a new `--config-upgrade` mode in `run-ha-soak.sh` (+ `SKILL.md` docs). Building and running this surfaced two real, independently-confirmed production defects: + - **bug-1468** (`cluster.c`): `myreq_handler`'s `OPCode_upgradeConfig` dispatch guard was inverted by an unrelated commit (`fd93997f`, 2026-05-31) — it rejected the only role (primary) that could ever legitimately reach it, making config upgrades unreachable from any network client at all. Fixed: restored correct polarity. + - **bug-1469** (`persist.c`): `apply_committed_entry`'s `WAL_CONFIG_MARKER` case never called `config_activate_committed()` — so the node that *originates* a config upgrade never refreshed its own in-memory config/queue state as part of committing it (only receiving standbys did, via a separate out-of-band push path). If that node was killed before some later, independent event happened to refresh it, its checkpoint froze with stale config content — a genuine, **live-reproduced** instance of bug-857's originally-hypothesized failure mode (task 8.5 could only prove a different, unit-level mechanism; this is the first real empirical case). Fixed: added `config_activate_committed(c)` inside the marker-apply case, positioned *before* the marker-triggered checkpoint write (checkpoint content is built from live runtime tables that only reflect the reload after activation runs, not from `myconf` directly) and guarded against `c->myconf == NULL` (a real crash risk hit in an existing test fixture). Fixing this also exposed and required fixing a genuine race in a pre-existing test (`test_commit_advance_retry_clears_push_before_commit_fence`) that the new activation side-effect made observable for the first time — narrowed the race window, did not weaken any assertion. Reviewer independently re-derived the lock-safety claim from scratch (traced every mutex site in `config_repl.c`, confirmed none reachable from `config_activate_committed`) and the thread-class claim (`apply_committed_entry` runs on a `VTASK_LIVE_BLOCK` pool worker, confirmed via `persist_apply_pump_kick`) rather than trusting the implementer's account — both held. + - **Still open, blocking a clean PASS — bug-1471**: a separate, much narrower 1-byte checkpoint divergence in a host-admin-status frame (not a config-activation issue), found independently by two concurrent sessions working this task and root-cause-not-yet-isolated. Also noted but deliberately **not** triggered by the current soak scenario: a deeper, genuinely separate gap where no live mechanism exists anywhere in this codebase to add a *new* queue to an already-running member's runtime `qmgr` without a restart (`config_activate_committed`/`load_primary_config` refresh `myconf->cluster_config` but nothing syncs that into the live `qmgr`) — the soak's upgrade-target config change was deliberately scoped to modifying an *existing* queue's field rather than adding one, specifically to test what Requirements 4.1-4.3/6.1/6.4 actually ask for without tripping over this separate, deeper architectural gap. Both are flagged in `.wolf/buglog.json` for whoever next picks up bug-1471 (or a dedicated follow-up for the queue-creation gap) — task 8.6's own checkbox stays unchecked until the soak reports a clean PASS. + - **Process note**: this task was interrupted by the same monthly-spend-limit condition hit earlier in this session (task 6.1), across three separate dispatch/resume cycles. Each resumption correctly checked live `git status`/`diff` before acting rather than blindly redoing work, and the final state was independently re-verified end-to-end (build + full 1025-test suite + file-integrity spot check) before being reviewed and committed — no corruption or duplicated work resulted, but it's worth flagging that this class of interruption can leave multiple agent instances transiently active in the same worktree. + - **Task 8.5 redirection (design.md's original mechanism was wrong; this is not the same as a review rejection):** the original hypothesis for bug-857's divergence — a job whose queue is removed by a config upgrade gets re-homed to `lost_and_found` on some members but not others, based on per-node restart timing — turned out to be **aspirational, never-implemented code**. Exhaustive grep across `job.c`/`queue.c`/`worker.c`/`config_repl.c` at both the pre-fix baseline commit (`e0c5e8b7`) and current HEAD confirmed: `create_lost_and_found` only ever creates the sentinel queue/host objects (and its entire body is `#ifndef XCE_TESTING`-gated); `restore_job`/`restore_job_array` never consult `qmgr`; `restore_queue_state` explicitly *skips* rather than re-homes when a queue is missing. `.wolf/buglog.json`'s bug-857 entry itself flags this: "identified via TLA+ gap analysis... NOT empirically reproduced yet." design.md's Requirements Traceability table assigned 6.2 to a live "HA soak new scenario" — also written against the same since-disproven hypothesis. Redirected (orchestrator decision, not a task-local implementer call) to a real, code-backed mechanism instead: `schedule_checkpoint`'s pre-existing, unmodified-by-this-spec WAL-record-count threshold trigger is genuinely decoupled from the config-marker `(term,lsn)` boundary, so two members' independently-timed checkpoints can capture different job/queue sets — the actual, general-purpose truth behind bug-857's claim ("no snapshot transfers an identical materialized state"). This is testable entirely within `controller_unit_test`, no live multi-process soak or old-commit worktree needed, since the threshold-trigger mechanism is still live, unmodified production code coexisting with the new marker-trigger at HEAD. Landed as `test_bug857_regression.c` (2 tests: divergence under threshold-only triggering vs. convergence under the real marker-triggered path, the latter under deliberately differing per-member processing pace to make the convergence claim non-trivial). Reviewer independently re-derived and confirmed the mechanism claim from first principles (not just re-running the tests) before approving — see `git blame`/review transcript for the full verification chain (schedule_checkpoint predates this spec via `git merge-base --is-ancestor`, zero lines of its body touched by tasks 1-7, `persist_apply_pump_run`'s strictly-sequential dispatch loop explains why the marker-triggered checkpoint's convergence isn't a timing coincidence). design.md's Requirements Traceability table (row 6.2) and Testing Strategy section were both corrected to match. **Heads-up for tasks 8.6-8.9**: they still legitimately need live multi-process HA-soak-level testing for Requirement 6.1 (chaotic multi-kill + config upgrade) — that infrastructure (a live config-upgrade CLI trigger doesn't exist anywhere in this codebase yet; confirmed via grep of `test.c`/`src/frontend`/`src/tools`) was NOT built by this task and will need to be built by whichever of 8.6-8.9 first needs to drive a live `OP_upgradeConfig` RPC against a running cluster. - **Task 8.4 round-1 rejection (real gap, caught by review):** the first pass built `configcompare_paths()` (correct, well-tested against `persist.c`/`config_repl.c`'s real `workdir/name/conf/current`+`conf/cluster.bin.` resolution convention) and linked `configcompare.c` into both `xce_test_src` and `controller_unit_test_src` in meson, but never registered it as a `cmd_configcompare` subcommand in `testsuite/test.c` — unlike its siblings `walcompare`/`snapverify`/`snapdiff`, each of which has a `cmd_*` wrapper in BOTH `top_commands[]` and `client_commands[]`. Left as delivered, the tool was a tested library function with no shell entry point — unusable by task 8.6/8.9's later HA soak scenarios, which design.md explicitly says will invoke `configcompare` directly against each node's data directory. Fixed with a 25-line, `test.c`-only remediation mirroring `cmd_snapdiff`'s exact shape; verified end-to-end via `echo 'configcompare ' | xce_test` (checked `$?` for both IDENTICAL/exit-0 and DIVERGENT/exit-1 cases, not just printed text). **Heads-up for any future task in this spec (or elsewhere) that adds a new offline testsuite tool in this family**: linking the `.c` file into the meson source lists is NOT sufficient — it must also get a `cmd_*` wrapper + `top_commands[]`/`client_commands[]` entries in `test.c`, or it silently has no way to be invoked outside the unit test binary. diff --git a/src/controller/cluster.c b/src/controller/cluster.c index 4cabe95..239f327 100644 --- a/src/controller/cluster.c +++ b/src/controller/cluster.c @@ -669,10 +669,30 @@ static int myreq_handler(vtask_t* task) break; case OPCode_upgradeConfig: #if CONF_ENABLE_UPGRADE_CONFIG - /* OPCode_upgradeConfig is sent BY the primary TO standbys to push config. - * Standbys must apply it; only a primary sending this to itself is invalid. */ - if (ha_is_primary(&c->ha)) { - /* Primary should not receive its own config push — ignore */ + /* OPCode_upgradeConfig is sent BY an admin client (or xceadmin's + * config-push tool) TO the primary controller, which decodes the + * blob, verifies it, and drives it through config_commit_change() + * (the (term,lsn) marker consensus path) — see + * do_upgrade_config_with_bytes's doc comment, config_repl.c. Only the + * primary can commit a WAL entry, so a standby cannot service this + * request; ha_dispatch_gate already denies OPCode_upgradeConfig for + * every non-primary role (it is absent from the CTRL_STANDBY / + * CTRL_INIT / CTRL_ELECTING allow-lists, ha.c), so this check is a + * defensive belt-and-suspenders guard, not the primary gate. + * + * bug-fix (2026-07-25): this check was inverted from + * commit fd93997f ("election-iomgr-refactor", 2026-05-31) onward — + * it rejected the request WHEN primary, unconditionally reachable + * only on primary (per the dispatch-gate note above), making + * handle_upgrade_config() permanently unreachable from any network + * caller (only unit tests calling it directly ever exercised it). + * Independently diagnosed and documented as a blocking risk by the + * admin-config-tool spec (design.md "Boundary Commitments" / + * "Risks") before config-upgrade-snapshot task 8.6 needed a live + * OP_upgradeConfig sender and hit the same wall. Restored to the + * pre-regression (correct) polarity. */ + if (!ha_is_primary(&c->ha)) { + /* Standby cannot commit a config change — reject loud. */ xce_reply_t ureply; init_reply_by_req(&ureply, &request); ureply.status = -EAGAIN; diff --git a/src/controller/persist.c b/src/controller/persist.c index ea0e822..8fb18d5 100644 --- a/src/controller/persist.c +++ b/src/controller/persist.c @@ -56,6 +56,20 @@ extern int restore_host_state(cluster_t* c, const char* name, * initialized default. */ extern int upgrade_materialization_verify(cluster_t* c); +/* config_activate_committed defined in config_repl.c (config-upgrade-snapshot + * task 5/6.2 remediation, bug-1469, Req 1.8/3.2/3.4) -- re-parses the + * locally-held config bytes referenced by the latest committed + * WAL_CONFIG_MARKER into c->myconf (load_primary_config), hash-verifying + * them first. Called unconditionally from apply_committed_entry's + * WAL_CONFIG_MARKER case below, BEFORE upgrade_materialization_verify: every + * member's own live apply of a marker it just committed must refresh its + * own c->myconf, not only members that separately receive an out-of-band + * handle_repl_config_push (see the call site's own doc comment for the full + * rationale). Declared here via extern, mirroring upgrade_materialization_ + * verify just above, rather than pulling in config_repl.h's full include + * surface for one symbol. */ +extern int config_activate_committed(cluster_t* c); + /* raft-apply-decoupling task 3.2: pending command-apply hand-off list node * (producer side; the not-yet-built ordered apply pump, task 6.1/6.2, is * the consumer). Declared here (rather than persist.h) so the list stays @@ -3807,6 +3821,102 @@ int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn, * checkpoint. `lsn` (this function's own parameter) is * already the marker's exact position, so it is passed * through explicitly instead of re-derived. */ + /* bug-1469 fix (live HA soak, config-upgrade-snapshot task 8.6): + * unconditionally re-run config activation right here, BEFORE the + * forced checkpoint below, so EVERY member -- primary, standby, and + * any future newly-elected leader -- is guaranteed to have + * refreshed c->myconf (and the runtime queue/host tables + * load_primary_config provisions from it) from disk BEFORE the + * checkpoint writer walks that same runtime state to build the + * queue-entity list. Before this call existed at all, + * config_activate_committed() was reachable from exactly two + * places: handle_repl_config_push (a STANDBY receiving a push) and + * the boot/recovery path -- never from a member's own live apply of + * the marker it just committed. A live 3-node chaotic-kill soak + * caught the gap empirically: the primary that ORIGINATED a config + * upgrade (do_upgrade_config_with_bytes, which saves the new bytes + * to disk and bumps config_generation but never re-activates them + * into c->myconf on itself) forced a checkpoint at the marker's own + * LSN whose queue-entity section still reflected the OLD + * (pre-upgrade) queue set, then was killed before any LATER, + * independent event (restart, or a racing inbound push) happened to + * call config_activate_committed -- freezing a stale + * queue-entity list into that checkpoint forever (a real + * cross-member materialized-state divergence, + * .wolf/buglog.json bug-1469's byte-level trace: the checkpoint's + * embedded raw config-bytes section correctly carried the new + * bytes, but the SEPARATE queue-entity list section -- built by + * walking runtime state, not the raw bytes -- did not, because that + * runtime state was never refreshed before the write). + * + * MUST run BEFORE persist_checkpoint_marker_triggered(), not merely + * before upgrade_materialization_verify: the checkpoint writer + * (write_snapshot -> collect_queue_frames/collect_host_frames) + * builds its queue/host-entity sections by walking c->qmgr/ + * c->workermgr -- LIVE RUNTIME TABLES, populated at boot from + * c->myconf->cluster_config and never otherwise re-synced from a + * later config change on this call path -- not by re-parsing + * mc->cluster_config directly. So activation must settle BEFORE + * persist_checkpoint_marker_triggered() runs, or the checkpoint + * this marker forces still captures whatever runtime state existed + * prior to activation, regardless of how soon afterward activation + * runs. (This ordering constraint is logically necessary given + * write_snapshot's own read path, confirmed by reading + * collect_queue_frames/write_snapshot directly -- not yet + * separately confirmed against a live soak re-run at the time this + * comment was written; see the task's own final live-soak + * verification step for that.) + * + * Symmetric with task 6.2's own fix shape below: one unconditional + * call at the single apply seam every member's marker-apply is + * guaranteed to pass through, rather than patching only the + * originating code path. + * + * Safe to call here for the same reason task 6.2's own call + * (upgrade_materialization_verify, below) is safe at this exact + * seam: neither config_activate_committed nor anything it + * transitively calls (config_committed_marker, local_config_sha256, + * conf_read_current_filename, conf_read_raw, load_primary_config) + * acquires apply_pump_lock, p->apply_pump_lock, or any other lock + * (verified -- config_repl.c's only two mutexes, c->ha.lock and + * c->persist.lock, are both taken in unrelated functions never on + * this call path) -- so no reentrant-lock hazard even though this + * runs while persist_apply_pump_run holds apply_pump_lock across + * its entire dispatch. The disk I/O this performs (WAL segment + * scan, config file read, Cap'n Proto decode) is the same class + * already unconditionally performed at this exact call site by + * upgrade_materialization_verify below (config_committed_marker's + * own scan + a config-bytes read/hash), so this adds no new + * blocking-I/O hazard class. Return value intentionally ignored + * (mirrors upgrade_materialization_verify below): a + * mismatch/absence return (-EAGAIN/-ENOENT) means this member's own + * serve-fence (config_serve_fenced, a SEPARATE mechanism from the + * Activation Fence below) is set instead, and that fence already + * has its own established retry path (config_schedule_activate_retry) + * driven off commit-advance -- this call's only job here is to give + * every marker-apply one unconditional attempt so a member never + * depends solely on a later, independent event to activate. + * + * Guarded on c->myconf != NULL: in production myconf is always + * wired during cluster init, well before any commit/apply can run, + * so this is a no-op check on every real code path -- but + * config_activate_committed's own -ENOENT fallback calls + * load_primary_config(c) unconditionally on c->myconf without its + * own null check (pre-existing in config_repl.c, out of this fix's + * scope to touch), and this call site is reachable from a + * bare/otherwise-empty test cluster_t that exercises + * WAL_CONFIG_MARKER apply mechanics without wiring myconf + * (test_upgrade_snapshot_apply.c's tusa_t fixture, + * test_marker_apply_via_live_pump_checkpoints_at_marker_own_lsn -- + * confirmed via a live SEGV in load_primary_config, address 0x28, + * before this guard was added) -- mirrors this same file's existing + * tolerance of a bare fixture elsewhere (e.g. collect_queue_frames' + * `c->qmgr == NULL` check, read_active_config_bytes' "no live + * cluster back-pointer" handling) rather than crashing a test that + * predates and is unrelated to config activation. */ + if (c->myconf != NULL) + (void)config_activate_committed(c); + int rc; #ifdef XCE_TESTING if (persist_marker_checkpoint_fault_hook != NULL && diff --git a/src/controller/testsuite/test.c b/src/controller/testsuite/test.c index 0161b0e..013fe1e 100644 --- a/src/controller/testsuite/test.c +++ b/src/controller/testsuite/test.c @@ -15,6 +15,7 @@ #include #include +#include /* SHA256() — cmd_upgradeconfig hashes the config blob */ #include "client.h" #include "config.h" /* XCE_MAXNAME_LEN */ @@ -122,6 +123,8 @@ extern int cmd_resval(int, char*[]); extern int cmd_jobkill(int, char*[]); extern int cmd_jobckpt(int, char*[]); extern int cmd_qctrl(int, char*[]); +extern int cmd_hostctrl(int, char*[]); +extern int cmd_upgradeconfig(int, char*[]); extern int cmd_runtasks(int, char*[]); extern int cmd_setpend(int, char*[]); extern int cmd_jobinfo(int, char*[]); @@ -230,6 +233,9 @@ static command_t client_commands[] = { { "reslist", cmd_reslist, "Display cluster resources" }, { "workerlist", cmd_workerlist, "list workers" }, { "qctrl", cmd_qctrl, "queue control" }, + { "hostctrl", cmd_hostctrl, "host control" }, + { "upgradeconfig", cmd_upgradeconfig, + "submit a config upgrade (cluster.bin file)" }, { "joblist", cmd_joblist, "list jobs" }, { "expect-jobs", cmd_expect_jobs, "assert current job count" }, { "expect-job", cmd_expect_job, "assert a JobId is present" }, @@ -1020,7 +1026,15 @@ int cmd_reslist(int argc, char* argv[]) static char* worker_status_str(uint32_t status) { - if ((status & XCE_WORKER_UNAVAIL) != 0) { + /* Admin-set CLOSED takes priority over the dynamic UNAVAIL bit: the + * config-upgrade-snapshot soak (task 8.6) needs to assert a host's + * administratively-set status specifically, and a host closed via + * hostctrl that later disconnects (e.g. its xce_test `worker`/`join` + * connection exits after the script ends) would otherwise print only + * "UNAVAIL", masking the admin bit this check exists to verify. */ + if ((status & XCE_WORKER_CLOSED) != 0) { + return "CLOSED"; + } else if ((status & XCE_WORKER_UNAVAIL) != 0) { return "UNAVAIL"; } else if ((status & XCE_WORKER_LOCKEDU) != 0) { return "LOCKU"; @@ -2224,6 +2238,139 @@ int cmd_qctrl(int argc, char* argv[]) return 0; } +/* hostctrl -o|-c -h — administratively open/close a host + * (xce_hostctrl_t{action, hostname} via xcecli_hostctrl). Mirrors cmd_qctrl's + * exact getopt shape; only -o/-c are wired (matching xce_hostctrl_req's own + * accepted action range, request.c — HOST_TAG/HOST_UNTAG exist in the enum + * but are rejected -EINVAL server-side today, so no flag is offered for them + * here). Used by the config-upgrade chaotic soak (task 8.6) to set a host's + * admin status before/around a config upgrade and assert it survives. */ +int cmd_hostctrl(int argc, char* argv[]) +{ + int opt; + int rc; + xce_hostctrl_t hostctrl; + int action = HOST_DO_NONE; + char* hostname = NULL; + + optind = 1; + + while ((opt = getopt(argc, argv, "och:")) != -1) { + switch (opt) { + case 'o': + action = HOST_OPEN; + break; + case 'c': + action = HOST_CLOSE; + break; + case 'h': + hostname = strdup(optarg); + break; + default: + break; + } + } + + if (hostname == NULL) { + fprintf(stderr, "you need to specify host name with -h\n"); + return -1; + } + + if (action == HOST_DO_NONE) { + fprintf(stderr, "you need to specify -o (open) or -c (close)\n"); + free(hostname); + return -1; + } + + hostctrl.action = action; + hostctrl.hostname = hostname; + + rc = xcecli_hostctrl(active_ctx->ctx, &hostctrl); + if (rc < 0) { + perror("xcecli_hostctrl"); + free(hostname); + return rc; + } + + printf("Operation on host %s done\n", hostname); + + free(hostname); + + return 0; +} + +/* upgradeconfig — submit a configuration upgrade + * (config-upgrade-snapshot task 8.6). Reads the given capnp-encoded + * ClusterConfig blob (the same format `xceadmin config build` produces and + * deploy-ha.sh installs as cluster.bin), computes its SHA-256 (OpenSSL + * SHA256(), matching config_repl.c's do_upgrade_config_with_bytes transit + * verification), and drives it through xcecli_upgradeconfig -> + * xce_upgradeconfig_req_raw (OP_upgradeConfig). Mirrors cmd_snapdiff's + * bounds-checked file-reading style. */ +int cmd_upgradeconfig(int argc, char* argv[]) +{ + if (argc != 2) { + fprintf(stderr, "usage: upgradeconfig \n"); + return -1; + } + + const char* path = argv[1]; + FILE* fp = fopen(path, "rb"); + if (fp == NULL) { + fprintf(stderr, "upgradeconfig: cannot open %s: %s\n", path, + strerror(errno)); + return -1; + } + + if (fseek(fp, 0, SEEK_END) != 0) { + fprintf(stderr, "upgradeconfig: fseek failed on %s: %s\n", path, + strerror(errno)); + fclose(fp); + return -1; + } + + long fsz = ftell(fp); + if (fsz <= 0) { + fprintf(stderr, "upgradeconfig: %s is empty or ftell failed\n", path); + fclose(fp); + return -1; + } + rewind(fp); + + uint8_t* bytes = (uint8_t*)malloc((size_t)fsz); + if (bytes == NULL) { + fprintf(stderr, "upgradeconfig: out of memory reading %s\n", path); + fclose(fp); + return -1; + } + + size_t nread = fread(bytes, 1, (size_t)fsz, fp); + fclose(fp); + if (nread != (size_t)fsz) { + fprintf(stderr, "upgradeconfig: short read on %s (%zu of %ld bytes)\n", + path, nread, fsz); + free(bytes); + return -1; + } + + uint8_t sha256[SHA256_DIGEST_LENGTH]; + SHA256(bytes, (size_t)fsz, sha256); + + int rc = xcecli_upgradeconfig(active_ctx->ctx, sha256, + sizeof(sha256), bytes, (size_t)fsz); + free(bytes); + + if (rc < 0) { + fprintf(stderr, "upgradeconfig: failed (rc=%d): %s\n", rc, + strerror(-rc)); + return rc; + } + + printf("Config upgrade submitted from %s (%ld bytes)\n", path, fsz); + + return 0; +} + int cmd_runtasks(int argc, char* argv[]) { xce_taskhost_t hosts[MAX_TASK_HOSTS]; diff --git a/src/controller/testsuite/test_config_repl.c b/src/controller/testsuite/test_config_repl.c index 6cf4b27..87e82d5 100644 --- a/src/controller/testsuite/test_config_repl.c +++ b/src/controller/testsuite/test_config_repl.c @@ -1819,9 +1819,25 @@ static void test_commit_advance_retry_clears_push_before_commit_fence(void) uint64_t tagN; { uint64_t t; memcpy(&t, shaN, 8); tagN = be64toh(t); } TEST_ASSERT_FALSE(memcmp(shaO, shaN, 32) == 0); - /* The NEW marker is APPENDED + flushed to the WAL (local tip advances). */ + /* The NEW marker is APPENDED + flushed to the WAL (local tip advances). + * bug-1469 fix interaction: flushing makes durable_lsn advance, and this + * single-member fixture's commit engine (majority=1) reacts to ANY + * durable advance by racing committed_lsn forward on its OWN background + * fsync-worker thread (commit_on_durable_advance) -- which, since + * apply_committed_entry's WAL_CONFIG_MARKER case now ALSO activates + * config (bug-1469), could apply-and-activate this NEW marker for real + * before this test's own thread gets a chance to pin committed_lsn back + * down below it. PIN IMMEDIATELY, right here, before any other work + * (previously this pin lived after conf_save_config below, leaving a + * multi-syscall window — several real fsyncs — for that background race + * to win against a single-member fixture's over-eager auto-commit, + * something a real multi-node standby never does since its + * committed_lsn is solely driven by the primary's commit-advance, never + * local durability). */ uint64_t new_mlsn = t43_append_marker_uncommitted(c, tagN, shaN); TEST_ASSERT_TRUE(new_mlsn > committed_at_old); + atomic_store_explicit(&c->persist.committed_lsn, committed_at_old, + memory_order_release); /* The OLD config is currently LIVE (activated in upgrade 1). Capture the live * decoded-config pointer as the activation witness: load_primary_config frees @@ -1837,9 +1853,10 @@ static void test_commit_advance_retry_clears_push_before_commit_fence(void) * commit barrier, which would advance committed_lsn to the durable WAL tip, and * it bumps the advertised config_version_tag to tagN (this is the very reason * the primary's config-behind re-push net does not re-fire — Finding D); we - * save FIRST, then PIN committed_lsn back below the NEW marker to model a - * standby whose committed prefix (driven by the primary's commit-advance, not - * local durability) has NOT yet advanced past the NEW marker. */ + * already pinned committed_lsn back below the NEW marker immediately above (to + * model a standby whose committed prefix, driven by the primary's commit- + * advance, not local durability, has NOT yet advanced past the NEW marker) — + * re-pin again here for defense-in-depth against the same race. */ TEST_ASSERT_EQUAL_INT(0, conf_save_config(mc, bytesN, blenN, shaN)); atomic_store_explicit(&c->persist.committed_lsn, committed_at_old, memory_order_release); @@ -2329,6 +2346,147 @@ static void test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss(voi printf("test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss: PASS\n"); } +/* ── bug-1469 regression: WAL_CONFIG_MARKER apply must ACTIVATE config on + * the ORIGINATING member, not just receiving standbys ────────────────────── + * + * Live 3-node chaotic-kill soak (config-upgrade-snapshot task 8.6, + * .wolf/buglog.json bug-1469): a config upgrade's WAL_CONFIG_MARKER commits + * and even checkpoints successfully on the ORIGINATING member (the primary + * that submitted the upgrade via do_upgrade_config_with_bytes), but + * apply_committed_entry's WAL_CONFIG_MARKER case (persist.c) never called + * config_activate_committed()/load_primary_config() -- so that member's + * c->myconf stayed parsed from the OLD config bytes until some LATER, + * independent event (a restart, or an incoming out-of-band + * handle_repl_config_push) happened to refresh it. The live soak killed the + * originating primary shortly after it committed the upgrade, before any + * such later event, freezing a stale c->myconf into its last checkpoint -- + * a genuine cross-member materialized-state divergence (ctrl-1's checkpoint + * ended up a different byte size than ctrl-2/ctrl-3's, both LSN 11). + * config_activate_committed WAS already being called correctly on + * RECEIVING standbys, via handle_repl_config_push -- this gap is specific + * to the member that originates/commits the marker on itself. + */ + +/* Build a valid ClusterConfig blob distinguishable by config_generation + + * clustername (mirrors t43_make_config_blob, parameterized so this test can + * commit two genuinely different, independently-decodable configs -- an + * "old" pre-upgrade config and a "new" post-upgrade config -- without + * relying on a byte-mutation of an opaque capnp blob, which risks + * corrupting the encoding; config-upgrade-snapshot's own standing gotcha + * notes t43_make_config_blob() returns byte-IDENTICAL content on repeat + * calls, so a genuinely different SECOND blob needs its own encode call). */ +static uint8_t *t1469_make_config_blob(uint64_t gen, const char *name, + size_t *len_out) +{ + cluster_config_t cfg; + struct capn cap; + struct capn_segment *cs; + ClusterConfig_ptr p; + uint8_t *buf; + size_t sz; + + memset(&cfg, 0, sizeof(cfg)); + cfg.clustername = (char *)name; + cfg.config_generation = gen; + + capn_init_malloc(&cap); + cs = capn_root(&cap).seg; + encode_ClusterConfig_ptr(cs, &p, &cfg); + capn_setp(capn_root(&cap), 0, p.p); + sz = (size_t)capn_size(&cap); + buf = (uint8_t *)malloc(sz); + TEST_ASSERT_NOT_NULL(buf); + capn_write_mem(&cap, buf, sz, 0); + capn_free(&cap); + *len_out = sz; + return buf; +} + +/* This test drives the upgrade marker through the REAL production apply + * path (config_append_marker + commit_submit + wait-for-applied, exactly + * like test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss + * above) on a single-member fixture acting as its OWN primary -- i.e. the + * ORIGINATING member applying its own committed marker, never receiving an + * out-of-band handle_repl_config_push. It NEVER calls + * config_activate_committed() for the upgrade marker itself, so the only + * way mc->cluster_config can reflect the new config is via + * apply_committed_entry's own WAL_CONFIG_MARKER dispatch. + * + * RED (before the fix): mc->cluster_config stays pinned to the OLD config + * (config_generation==1, "t1469-old") after the new marker's real, + * successful commit+apply -- proving the marker-triggered dispatch never + * activates config on the member that originated it. + * GREEN (after the fix): mc->cluster_config reflects the NEW config + * (config_generation==2, "t1469-new") immediately after the marker's apply + * returns, with no restart and no out-of-band push. */ +static void test_wal_config_marker_apply_activates_originating_member_config(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_cr1469_activate_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + cluster_t *c = t42_make_cluster(tmp); + TEST_ASSERT_NOT_NULL(c); + t42_make_leader(c, /*term*/69, /*n_peers*/1); + TEST_ASSERT_EQUAL_INT(0, + commit_engine_init(&c->commit, &c->persist, 1, 8)); + repl_set_commit_engine(&c->persist.repl, &c->commit); + myconf_t *mc = t43_attach_myconf(c, tmp); + + /* Establish the pre-upgrade steady state: an OLD config already + * committed and ACTIVATED via the legitimate, already-tested + * config_activate_committed path (mirrors 4.3-a above) -- this member + * is already running under config_generation=1 before any upgrade is + * ever submitted, exactly like a real cluster member. */ + size_t oldlen = 0; + uint8_t *oldbytes = t1469_make_config_blob(1, "t1469-old", &oldlen); + uint8_t oldsha[32]; + SHA256(oldbytes, oldlen, oldsha); + uint64_t oldtag; { uint64_t t; memcpy(&t, oldsha, 8); oldtag = be64toh(t); } + TEST_ASSERT_EQUAL_INT(0, conf_save_config(mc, oldbytes, oldlen, oldsha)); + free(oldbytes); + t43_commit_marker(c, oldtag, oldsha); + TEST_ASSERT_EQUAL_INT(0, config_activate_committed(c)); + TEST_ASSERT_NOT_NULL(mc->cluster_config); + TEST_ASSERT_EQUAL_UINT64(1, mc->cluster_config->config_generation); + + /* Now the upgrade: NEW config bytes saved locally exactly as + * do_upgrade_config_with_bytes's own step 5 (conf_save_config) does on + * the ORIGINATING primary, then a NEW marker committed through the + * exact same real path bug-1469's live soak exercised. Deliberately + * NEVER calls config_activate_committed() for this marker anywhere + * below -- activation must come from apply_committed_entry's own + * dispatch, or this assertion fails. */ + size_t newlen = 0; + uint8_t *newbytes = t1469_make_config_blob(2, "t1469-new", &newlen); + uint8_t newsha[32]; + SHA256(newbytes, newlen, newsha); + uint64_t newtag; { uint64_t t; memcpy(&t, newsha, 8); newtag = be64toh(t); } + TEST_ASSERT_EQUAL_INT(0, conf_save_config(mc, newbytes, newlen, newsha)); + free(newbytes); + t43_commit_marker(c, newtag, newsha); + + /* THE ASSERTION THAT PROVES THE FIX: mc->cluster_config must now + * reflect the NEW config -- purely from apply_committed_entry's own + * WAL_CONFIG_MARKER dispatch, exactly the production path the + * ORIGINATING primary runs on itself. Before the fix this stays pinned + * at generation 1 ("t1469-old") forever -- exactly bug-1469's observed + * cross-member divergence, reproduced at the unit level. */ + TEST_ASSERT_NOT_NULL(mc->cluster_config); + TEST_ASSERT_EQUAL_UINT64_MESSAGE(2, mc->cluster_config->config_generation, + "bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never " + "activated the new config on the ORIGINATING member -- c->myconf " + "stayed pinned to the OLD config"); + TEST_ASSERT_EQUAL_STRING("t1469-new", mc->cluster_config->clustername); + TEST_ASSERT_EQUAL_HEX64(newtag, mc->config_version_tag); + TEST_ASSERT_EQUAL_INT(0, config_serve_fenced(c)); + + t43_free_myconf(c); + commit_engine_fini(&c->commit); + t42_teardown(c); + printf("test_wal_config_marker_apply_activates_originating_member_config: PASS\n"); +} + /* ── wait_all_with_timeout (XCE_TESTING shim) ───── */ #ifdef XCE_TESTING @@ -5144,6 +5302,9 @@ void test_config_repl(void) * production trigger (apply_committed_entry's WAL_CONFIG_MARKER case), * not a direct test call -- proves the fence is reachable at all. */ RUN_TEST(test_wal_config_marker_apply_sets_fence_via_real_checkpoint_miss); + /* bug-1469: WAL_CONFIG_MARKER apply must activate config on the + * ORIGINATING member too, not just receiving standbys. */ + RUN_TEST(test_wal_config_marker_apply_activates_originating_member_config); #ifdef XCE_TESTING RUN_TEST(test_load_primary_config_missing_returns_error); RUN_TEST(test_wait_all_with_timeout_slow_threads); diff --git a/src/include/client.h b/src/include/client.h index ba178ff..1de3ac5 100644 --- a/src/include/client.h +++ b/src/include/client.h @@ -95,6 +95,14 @@ extern int xcecli_jobctrl(xcecli_ctx_t, xce_jobctrl_t*); extern int xcecli_notify_jobchkpt(xcecli_ctx_t, uint64_t, char*); extern int xcecli_queuectrl(xcecli_ctx_t, xce_queuectrl_t*); extern int xcecli_hostctrl(xcecli_ctx_t, xce_hostctrl_t*); +/* xcecli_upgradeconfig: submit a configuration upgrade (sha256 + serialised + * ClusterConfig blob) to the primary via OP_upgradeConfig, transparently + * surviving a primary change mid-call via XCECLI_FAILOVER_CALL (same + * write-op failover contract as xcecli_hostctrl/xcecli_queuectrl). Returns + * the server's reply status (0 on success). */ +extern int xcecli_upgradeconfig(xcecli_ctx_t, const uint8_t *sha256, + size_t sha256_len, const uint8_t *bytes, + size_t bytes_len); extern int xcecli_runtask(xcecli_ctx_t, xce_taskrun_t*); extern int xcecli_notify_taskstart(xcecli_ctx_t, xce_task_pidinfo_t*); extern int xcecli_notify_taskfail(xcecli_ctx_t, xce_task_failinfo_t*); diff --git a/src/include/request.h b/src/include/request.h index 14320dd..d2f4ef0 100644 --- a/src/include/request.h +++ b/src/include/request.h @@ -185,6 +185,19 @@ extern int xce_upgrade_config_req_extract_raw(const uint8_t *pkt_data, size_t *out_len, uint8_t **out_sha256); +/* + * xce_upgradeconfig_req_raw: send OP_upgradeConfig carrying raw byte arrays + * (sha256 + the serialised ClusterConfig blob). Bypasses the generated + * encode_UpgradeConfigRequest path (a no-op for the sha256/bytes Data fields + * — same class of limitation as xce_replconfigpush_req_raw) by constructing + * the capnp message directly via capn_new_list8 + capn_setv8 + write_Request. + * Ordinary CLIENT request: seq/from are populated the same way xce_request() + * populates them for every other client op. + */ +extern int xce_upgradeconfig_req_raw(endpoint_t *ep, + const uint8_t *sha256, size_t sha256_len, + const uint8_t *bytes, size_t bytes_len); + /* ── Client-Side Transparent Retry tunables (Req 6.1, 6.2) ────────────────── * * xce_retry_config_t: configurable retry parameters with safe production diff --git a/src/include/xceapi.h b/src/include/xceapi.h index 7786183..e4f93ae 100644 --- a/src/include/xceapi.h +++ b/src/include/xceapi.h @@ -28,6 +28,17 @@ extern int xce_update_resvals(xce_ctx_t*, int, xce_resval_t**); extern int xce_jobctrl(xce_ctx_t*, xce_jobctrl_t*); extern int xce_queuectrl(xce_ctx_t*, xce_queuectrl_t*); extern int xce_hostctrl(xce_ctx_t*, xce_hostctrl_t*); +/* xce_upgradeconfig: send OP_upgradeConfig (sha256 + serialised ClusterConfig + * blob) via xce_upgradeconfig_req_raw and wait for the reply. Returns the + * server's reply status (0 on success; a negative errno/E_XCE_* code on + * failure — the server logs the human-readable reason, see + * do_upgrade_config_with_bytes/send_upgrade_error, config_repl.c). Mirrors + * xce_hostctrl/xce_queuectrl's shape; the richer UpgradeConfigReply fields + * (version_tag/acked_count/error text) are not surfaced to the caller — no + * consumer of this API currently needs them beyond success/failure. */ +extern int xce_upgradeconfig(xce_ctx_t*, const uint8_t *sha256, + size_t sha256_len, const uint8_t *bytes, + size_t bytes_len); extern int xce_runtask(xce_ctx_t*, xce_taskrun_t*); extern int xce_update_jusage(xce_ctx_t*, uint64_t, int, int*, int, int*, int, xce_rusage_t**); diff --git a/src/lib/client.c b/src/lib/client.c index cde3824..c0b8223 100644 --- a/src/lib/client.c +++ b/src/lib/client.c @@ -778,6 +778,25 @@ int xcecli_hostctrl(xcecli_ctx_t c, xce_hostctrl_t* ctrlinfo) return rc; } +int xcecli_upgradeconfig(xcecli_ctx_t c, const uint8_t *sha256, + size_t sha256_len, const uint8_t *bytes, + size_t bytes_len) +{ + api_ctx_t* ctx = (api_ctx_t*)c; + int rc; + + if ((ctx == NULL) || (sha256 == NULL) || (sha256_len == 0) + || (bytes == NULL) || (bytes_len == 0)) { + return -EINVAL; + } + + rc = XCECLI_FAILOVER_CALL(ctx, /*is_write=*/true, + xce_upgradeconfig(&(ctx->ctx), sha256, + sha256_len, bytes, + bytes_len)); + return rc; +} + int xcecli_runtask(xcecli_ctx_t c, xce_taskrun_t* info) { api_ctx_t* ctx = (api_ctx_t*)c; diff --git a/src/lib/request.c b/src/lib/request.c index df8a063..0036f34 100644 --- a/src/lib/request.c +++ b/src/lib/request.c @@ -499,6 +499,94 @@ int xce_replconfigpush_req_raw(endpoint_t *ep, xce_peerinfo_t *from, return (enq_rc < 0) ? enq_rc : 0; } +/* + * xce_upgradeconfig_req_raw: send OP_upgradeConfig with raw byte arrays. + * + * Bypasses encode_UpgradeConfigRequest (a literal no-op for the sha256/bytes + * Data fields — see the doc comment above upgrade_config_req_s in typemap.h; + * confirmed by inspection of the generated xce.build/lib/controller.capnp.c). + * Instead, constructs the struct Request / struct UpgradeConfigRequest + * directly using the capnp wire-format API (capn_new_list8 + capn_setv8), + * mirroring xce_replconfigpush_req_raw's exact bypass pattern. + * + * Unlike that controller-to-controller helper, this is an ordinary CLIENT + * request: seq and from are populated the same way the generic xce_request() + * populates them for every other client op (endpoint_getseq(ep), ep->peer), + * not passed in by the caller. + */ +int xce_upgradeconfig_req_raw(endpoint_t *ep, + const uint8_t *sha256, size_t sha256_len, + const uint8_t *bytes, size_t bytes_len) +{ + struct capn c; + struct capn_segment *cs; + struct Request rq; + struct UpgradeConfigRequest up_req; + UpgradeConfigRequest_ptr up_ptr; + Request_ptr rp; + io_packet_t *pkt; + int rc; + + if (ep == NULL) + return -EINVAL; + + if (sha256 == NULL || sha256_len == 0 || bytes == NULL || bytes_len == 0) + return -EINVAL; + + capn_init_malloc(&c); + cs = capn_root(&c).seg; + + memset(&rq, 0, sizeof(rq)); + memset(&up_req, 0, sizeof(up_req)); + + /* sha256: write as a List(UInt8) — capn_list8 and capn_data share {capn_ptr p} */ + { + capn_list8 sha_list = capn_new_list8(cs, (int)sha256_len); + capn_setv8(sha_list, 0, sha256, (int)sha256_len); + up_req.sha256.p = sha_list.p; + } + + /* config bytes */ + { + capn_list8 bytes_list = capn_new_list8(cs, (int)bytes_len); + capn_setv8(bytes_list, 0, bytes, (int)bytes_len); + up_req.bytes.p = bytes_list.p; + } + + up_ptr = new_UpgradeConfigRequest(cs); + write_UpgradeConfigRequest(&up_req, up_ptr); + + rq.opcode = OPCode_upgradeConfig; + rq.seq = endpoint_getseq(ep); + rq.body_which = Request_body_upgradeconfigrequestmsg; + rq.body.upgradeconfigrequestmsg = up_ptr; + encode_PeerInfo_ptr(cs, &rq.from, ep->peer); + + rp = new_Request(cs); + write_Request(&rq, rp); + capn_setp(capn_root(&c), 0, rp.p); + + pkt = (io_packet_t *)calloc(1, sizeof(*pkt)); + if (!pkt) { + capn_free(&c); + return -ENOMEM; + } + + pkt->sz = capn_size(&c); + rc = alloc_packet(pkt); + if (rc < 0) { + free(pkt); + capn_free(&c); + return -errno; + } + + capn_write_mem(&c, (uint8_t *)pkt->data, pkt->sz, 0); + capn_free(&c); + + int enq_rc = endpoint_enq(ep, &(pkt->link)); + return (enq_rc < 0) ? enq_rc : 0; +} + /* wal_entry_to_wire: fill a wire WalCatchupEntry from a C entry's header * fields and raw payload bytes. Sub-entry nesting is one level deep — * elements written here never carry their own entries list. */ diff --git a/src/lib/xceapi.c b/src/lib/xceapi.c index c4cb781..02933c5 100644 --- a/src/lib/xceapi.c +++ b/src/lib/xceapi.c @@ -647,6 +647,36 @@ int xce_hostctrl(xce_ctx_t* ctx, xce_hostctrl_t* ctrl) return status; } +int xce_upgradeconfig(xce_ctx_t* ctx, const uint8_t *sha256, + size_t sha256_len, const uint8_t *bytes, + size_t bytes_len) +{ + int rc; + int status; + + if (ctx == NULL) { + return -EINVAL; + } + + if (ctx->state != XCE_CTX_RUN) { + return -EINVAL; + } + + rc = xce_upgradeconfig_req_raw(ctx->ep, sha256, sha256_len, bytes, + bytes_len); + if (rc < 0) { + return rc; + } + + struct timespec _dl; cbuf_deadline_10s(&_dl); + rc = cbuf_timedget(&(ctx->wbuf), &status, &_dl); + if (rc < 0) { + return rc; + } + + return status; +} + int xce_runtask(xce_ctx_t* ctx, xce_taskrun_t* runinfo) { int rc;