feat(resource-value-verification): worker join and resource injection (task 6.1)
inject_resources() joins a synthetic worker through xce_test's worker
context and runs resupdate for the requested resource=value pairs
(default: Requirement 1's numeric gap set + mem/cpuf regression names,
deliberately excluding type/model due to bug-1574's live segfault risk).
Per-name accept/reject results captured into INJECTED_VALUES/
INJECTED_RESULTS plus injection-results.tsv for tasks 6.2-6.4.
Also suppresses the real xcecd_virtual worker daemon that deploy-ha.sh's
pre-existing worker branch would otherwise silently start whenever the
build dir contains one (XCE_WORKER_BIN=/bin/true), which would have
violated Requirement 2.1 - bug-1577.
Blocked pending a separate prerequisite fix (bug-1578, commit b406dd7):
live testing surfaced a pre-existing controller crash on worker disconnect,
unrelated to this feature, now fixed with regression coverage. Live-
reconfirmed post-fix: controller survives the full join+resupdate+quit flow
indefinitely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
b406dd762a
commit
75a0839d47
SSH key fingerprint:
SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
2 changed files with 198 additions and 29 deletions
|
|
@ -5,17 +5,30 @@
|
|||
# resource value via xce_test's resupdate, confirm it is visible at the
|
||||
# controller, live scheduler, and Go apiserver).
|
||||
#
|
||||
# resource-value-verification spec, task 5.1 SCOPE: this file currently
|
||||
# implements ONLY the deployment step -- stand up a fresh controller +
|
||||
# scheduler (-vvv) + apiserver (debug logging) deployment via
|
||||
# xce-deploy-ha's deploy-ha.sh + generated start-all.sh, confirm all three
|
||||
# components are actually up, and clean up (trap-based, covering a
|
||||
# mid-deployment abort, not just a completed run). Injection (task 6.1),
|
||||
# per-layer verification (tasks 6.2-6.4), and verdict aggregation /
|
||||
# resource-value-verification spec, task 5.1+6.1 SCOPE: this file currently
|
||||
# implements deployment (task 5.1) PLUS worker join + resource injection
|
||||
# (task 6.1) -- stand up a fresh controller + scheduler (-vvv) + apiserver
|
||||
# (debug logging) deployment via xce-deploy-ha's deploy-ha.sh + generated
|
||||
# start-all.sh, confirm all three components are actually up, join a
|
||||
# SYNTHETIC worker (no real worker daemon process) through xce_test's worker
|
||||
# context, run resupdate for one or more resource=value pairs, and clean up
|
||||
# (trap-based, covering a mid-deployment abort, not just a completed run).
|
||||
# Per-layer verification (tasks 6.2-6.4) and verdict aggregation /
|
||||
# cleanup-on-every-path (task 7.1) are added by LATER tasks -- see the
|
||||
# "Extension points" markers in main() below for exactly where they hook in.
|
||||
# See .kiro/specs/resource-value-verification/{requirements,design}.md,
|
||||
# Requirements 5.3, 6.1, 6.5 for this task; the full skill covers 3.1-6.5.
|
||||
# Requirements 2.1, 2.2, 5.3, 6.1, 6.4, 6.5 for these tasks; the full skill
|
||||
# covers 3.1-6.5.
|
||||
#
|
||||
# Task 6.1 deliberate deviation from design.md: the default (no-args)
|
||||
# resource set is Requirement 1's gap-resource set (r15m, pg, ls, it, swp,
|
||||
# io, tmp) plus a couple of already-supported NUMERIC names for regression
|
||||
# coverage (mem, cpuf) -- NOT "type"/"model" as design.md's illustrative text
|
||||
# suggests. bug-1574 (xce_test can SEGFAULT, non-deterministically, on
|
||||
# `resupdate -n type/-n model ...` -- uninitialized-memory UB in
|
||||
# set_resval()'s Value_u_svalue fallback path) makes including a string-typed
|
||||
# resource in this script's own default live-testing path a real crash risk,
|
||||
# not a theoretical one. Revisit once bug-1574 is fixed.
|
||||
#
|
||||
# Must be invoked under an environment with `nix` on PATH (same as
|
||||
# xce-deploy-ha / xce-test-ha) -- this script shells out to `nix develop`
|
||||
|
|
@ -38,16 +51,22 @@
|
|||
# convention. $XCE_RESVERIFY_TOP overrides the default
|
||||
# when --dir is not given (same precedence pattern as
|
||||
# xce-test-ha's $XCE_HA_TOP).
|
||||
# resource=value NOT YET IMPLEMENTED (task 6.1 -- worker join +
|
||||
# injection). Accepted and parsed here so argument
|
||||
# handling doesn't need reshaping later, but currently
|
||||
# unused by this task's deployment-only scope.
|
||||
# resource=value One or more resources to inject via resupdate, e.g.
|
||||
# `r15m=1.5 pg=3`. May repeat. When none are given,
|
||||
# defaults to Requirement 1's gap-resource set plus a
|
||||
# couple of already-numeric names for regression
|
||||
# coverage (see the top-of-file bug-1574 note for why
|
||||
# the default deliberately excludes type/model).
|
||||
#
|
||||
# Exit 0 == this task's scope succeeded (deployment stood up, all three
|
||||
# components confirmed up, cleaned up per --keep-running). Non-zero ==
|
||||
# deployment failed at some step (see the printed "deployment failed at
|
||||
# step X" message). Once tasks 6.1-7.1 land, the exit code instead reflects
|
||||
# the injection/verification PASS/FAIL verdict.
|
||||
# components confirmed up, worker joined, resupdate ran for every requested
|
||||
# resource -- accepted AND rejected outcomes are both a successful run of
|
||||
# THIS script's scope, see inject_resources() -- cleaned up per
|
||||
# --keep-running). Non-zero == deployment failed at some step, the worker
|
||||
# failed to join, or resupdate produced no observable accept/reject outcome
|
||||
# for some name (a structural injection failure, distinct from a legitimate
|
||||
# rejection). Once tasks 6.2-7.1 land, the exit code instead reflects the
|
||||
# full injection/verification PASS/FAIL verdict.
|
||||
#
|
||||
set -u
|
||||
|
||||
|
|
@ -59,6 +78,10 @@ BUILD_DIR="${XCE_RESVERIFY_BUILD_DIR:-$REPO/src/xce.build}"
|
|||
BINARY="${XCE_RESVERIFY_BIN:-$BUILD_DIR/controller/xce_controller}"
|
||||
SCHED_BIN="${XCE_RESVERIFY_SCHED_BIN:-$BUILD_DIR/scheduler/xce_scheduler}"
|
||||
APISERVER_BIN="${XCE_RESVERIFY_APISERVER_BIN:-/tmp/apiserver}"
|
||||
# xce_test -- drives injection (task 6.1) via its worker-context `join`/
|
||||
# `resupdate` commands (same binary xce-test-ha's run-ha-soak.sh uses, same
|
||||
# build dir as $BINARY since both come from the controller meson target).
|
||||
XT="${XCE_RESVERIFY_XT:-$BUILD_DIR/controller/xce_test}"
|
||||
|
||||
# Single-node cluster, deliberately -- this skill verifies resource-value
|
||||
# PROPAGATION (controller -> scheduler -> apiserver), not raft/HA failover
|
||||
|
|
@ -84,8 +107,10 @@ Usage: $0 [--keep-running] [--dir DIR] [resource=value ...]
|
|||
always preserved either way.
|
||||
--dir DIR scratch/deploy directory (default: fresh mktemp -d, or
|
||||
\$XCE_RESVERIFY_TOP).
|
||||
resource=value reserved for task 6.1 (injection); accepted but not yet
|
||||
acted on by this task's deployment-only scope.
|
||||
resource=value inject this resource via resupdate (may repeat). Default
|
||||
when none given: r15m=1.5 pg=2.5 ls=3.5 it=4.5 swp=5.5
|
||||
io=6.5 tmp=7.5 mem=8.5 cpuf=9.5 (numeric-only -- see
|
||||
bug-1574 note at top of file).
|
||||
EOF
|
||||
}
|
||||
|
||||
|
|
@ -200,6 +225,7 @@ precondition_checks() {
|
|||
[ -x "$BINARY" ] || fail "controller binary missing: $BINARY (run /xce-build controller)"
|
||||
[ -x "$SCHED_BIN" ] || fail "scheduler binary missing: $SCHED_BIN (run /xce-build scheduler) -- required, this skill verifies the scheduler layer"
|
||||
[ -x "$APISERVER_BIN" ] || fail "apiserver binary missing: $APISERVER_BIN (cd src/frontend && make build-server, or the CGO-flags workaround -- see tasks.md's Implementation Notes) -- required, this skill verifies the apiserver layer"
|
||||
[ -x "$XT" ] || fail "xce_test binary missing: $XT (run /xce-build controller) -- required to join a synthetic worker and run resupdate"
|
||||
command -v nix >/dev/null 2>&1 || fail "nix not found on PATH -- required to resolve the deploy-environment bash and invoke deploy-ha.sh"
|
||||
}
|
||||
|
||||
|
|
@ -221,7 +247,26 @@ deploy() {
|
|||
|
||||
# ── Step 2: start-all.sh -- controller -> health-check -> scheduler/apiserver
|
||||
start_all() {
|
||||
log "invoking start-all.sh (controller, scheduler at -vvv, apiserver with debug logging)"
|
||||
# deploy-ha.sh's Step 3 also generates + start-all.sh also launches a
|
||||
# REAL worker daemon (xcecd_virtual) whenever one happens to exist
|
||||
# alongside $BINARY in the build dir (its own pre-existing, opt-in-by-
|
||||
# presence worker/scheduler convention, predating this feature -- see
|
||||
# task 3.1's notes). This skill's whole point (Requirement 2.1) is that
|
||||
# NO real worker daemon process is needed -- task 6.1 injects via a
|
||||
# SYNTHETIC xce_test-joined worker instead -- and a real worker sitting
|
||||
# in the background for this run's whole duration is worse than just
|
||||
# off-contract: it periodically re-reports its OWN resources (~every
|
||||
# 15s), and a resource's SECOND periodic report is bug-1572 (open,
|
||||
# unfixed scheduler double-free/crash) -- a real background worker is a
|
||||
# live crash risk to this run, not just noise. start-worker.sh reads
|
||||
# $XCE_WORKER_BIN itself (falling back to the real xcecd_virtual path)
|
||||
# -- point it at a real no-op binary so start-all.sh's worker step exits
|
||||
# immediately instead of leaving a daemon running; deploy-ha.sh itself
|
||||
# is untouched (out of boundary), this only affects what THIS skill's
|
||||
# own start-all.sh invocation actually launches.
|
||||
export XCE_WORKER_BIN="/bin/true"
|
||||
|
||||
log "invoking start-all.sh (controller, scheduler at -vvv, apiserver with debug logging; real worker daemon suppressed via XCE_WORKER_BIN=/bin/true -- see comment above)"
|
||||
if ! "$BASH_BIN" "$TOP/scripts/start-all.sh"; then
|
||||
fail "deployment failed at step start-all.sh (nonzero exit)"
|
||||
fi
|
||||
|
|
@ -267,6 +312,130 @@ confirm_deployment_up() {
|
|||
log "all three components confirmed up under $TOP"
|
||||
}
|
||||
|
||||
# ── Step 4: join a synthetic worker + inject resource values (task 6.1) ────
|
||||
# Requirement 1's gap-resource set (numeric) plus a couple of already-numeric
|
||||
# names for regression coverage -- see the top-of-file bug-1574 note for why
|
||||
# this deliberately excludes type/model (design.md's own illustrative text
|
||||
# names them, but xce_test can SEGFAULT on a string-typed resupdate, so this
|
||||
# is a documented, deliberate deviation, not an oversight).
|
||||
DEFAULT_RESOURCE_ARGS=(
|
||||
r15m=1.5 pg=2.5 ls=3.5 it=4.5 swp=5.5 io=6.5 tmp=7.5
|
||||
mem=8.5 cpuf=9.5
|
||||
)
|
||||
|
||||
# Fixed synthetic worker identity for this run -- tags every resupdate'd
|
||||
# value (Requirement 2.2), exactly like `set myname <host>` + `worker` +
|
||||
# `join` in xce-test-ha's run-ha-soak.sh (identical established pattern).
|
||||
# A fresh $TOP per invocation means no cross-run collision risk.
|
||||
WORKER_HOST="resverify-worker"
|
||||
|
||||
declare -A INJECTED_VALUES=() # resource name -> injected value, as sent
|
||||
declare -A INJECTED_RESULTS=() # resource name -> "accepted" | "rejected: <reason>" | "rejected: no result line observed"
|
||||
|
||||
# Run a batch of xce_test commands (stdin script). Two things this MUST do,
|
||||
# both hard-won (see cerebrum.md / buglog.json):
|
||||
# - ASAN_OPTIONS=detect_leaks=0: xce_test is an ASAN build with small
|
||||
# pre-existing intentional REPL-lifetime leaks; its default leak-detect
|
||||
# exit path can swallow buffered stdout on a piped/redirected run,
|
||||
# losing the very accept/reject lines this function parses.
|
||||
# - wrapped in `nix develop` (like deploy()'s own BASH_BIN resolution
|
||||
# above): unlike $BINARY/$SCHED_BIN/$APISERVER_BIN (started via
|
||||
# generated scripts whose binaries carry a baked RPATH), xce_test's own
|
||||
# runtime linkage has not been confirmed independent of the nix
|
||||
# devShell's library path -- wrap defensively, mirroring deploy()'s own
|
||||
# "shell out to nix develop for the steps that need it" posture.
|
||||
xtest() {
|
||||
nix develop "$REPO/src" --command env ASAN_OPTIONS=detect_leaks=0 timeout 60 "$XT" 2>&1
|
||||
}
|
||||
|
||||
# inject_resources [resource=value ...]
|
||||
# No args -> DEFAULT_RESOURCE_ARGS. Joins one synthetic worker (no real
|
||||
# worker daemon process anywhere in this flow -- Requirement 2.1) and runs
|
||||
# ONE resupdate invocation carrying every pair, so task 2.2's per-name
|
||||
# accept/reject accounting covers the whole batch in a single RPC round
|
||||
# trip. Populates INJECTED_VALUES/INJECTED_RESULTS (and $TOP/logs/
|
||||
# injection-results.tsv) for tasks 6.2-6.4's extension points to consume.
|
||||
inject_resources() {
|
||||
local pairs=("$@")
|
||||
local pair name value resupdate_line out result
|
||||
|
||||
if [ "${#pairs[@]}" -eq 0 ]; then
|
||||
pairs=("${DEFAULT_RESOURCE_ARGS[@]}")
|
||||
fi
|
||||
|
||||
resupdate_line=""
|
||||
for pair in "${pairs[@]}"; do
|
||||
name="${pair%%=*}"
|
||||
value="${pair#*=}"
|
||||
if [ -z "$name" ] || [ "$name" = "$pair" ] || [ -z "$value" ]; then
|
||||
fail "malformed resource argument (expected resource=value): $pair"
|
||||
fi
|
||||
INJECTED_VALUES["$name"]="$value"
|
||||
resupdate_line="$resupdate_line -n $name -v $value"
|
||||
done
|
||||
|
||||
log "joining synthetic worker '$WORKER_HOST' (masters=$PEERS) and injecting: ${pairs[*]}"
|
||||
|
||||
local script
|
||||
script="$(printf 'set masters %s\nset myname %s\nworker\njoin\nresupdate%s\nquit\n' \
|
||||
"$PEERS" "$WORKER_HOST" "$resupdate_line")"
|
||||
|
||||
out="$(printf '%s' "$script" | xtest)"
|
||||
|
||||
printf '%s\n' "$out" | grep -Fxq -- "host join ok" \
|
||||
|| fail "worker join failed -- no synthetic worker daemon process exists so this is the whole injection path; xce_test output:
|
||||
$out"
|
||||
|
||||
for pair in "${pairs[@]}"; do
|
||||
name="${pair%%=*}"
|
||||
if printf '%s\n' "$out" | grep -Fxq -- "${name}: accepted"; then
|
||||
result="accepted"
|
||||
elif printf '%s\n' "$out" | grep -Fxq -- "${name}: rejected: unknown resource"; then
|
||||
result="rejected: unknown resource"
|
||||
elif printf '%s\n' "$out" | grep -Fxq -- "${name}: rejected: value mismatch"; then
|
||||
result="rejected: value mismatch"
|
||||
else
|
||||
result="rejected: no result line observed"
|
||||
fi
|
||||
INJECTED_RESULTS["$name"]="$result"
|
||||
log "resupdate: $name=${INJECTED_VALUES[$name]} -> $result"
|
||||
done
|
||||
|
||||
# Manifest for post-run inspection and for tasks 6.2-6.4 to consume
|
||||
# (this same script process already has INJECTED_VALUES/INJECTED_RESULTS
|
||||
# in memory; the file is a durable, human-inspectable mirror of the
|
||||
# same data, alongside the always-preserved logs/ convention).
|
||||
mkdir -p "$TOP/logs"
|
||||
{
|
||||
for pair in "${pairs[@]}"; do
|
||||
name="${pair%%=*}"
|
||||
printf '%s\t%s\t%s\n' "$name" "${INJECTED_VALUES[$name]}" "${INJECTED_RESULTS[$name]}"
|
||||
done
|
||||
} > "$TOP/logs/injection-results.tsv"
|
||||
|
||||
# This flow never starts a real worker daemon (xcecd/xcecd_virtual) --
|
||||
# structurally guaranteed by construction (the only process this
|
||||
# function starts is xce_test itself, and only its worker CONTEXT, not
|
||||
# a separate daemon binary). Confirm explicitly rather than just assert
|
||||
# it in a comment, since Requirement 2.1/6.4's whole point is "no real
|
||||
# worker daemon process required".
|
||||
if pgrep -f -- 'xcecd_virtual|/xcecd( |$)' >/dev/null 2>&1; then
|
||||
log "note: a process matching xcecd/xcecd_virtual is running on this machine -- NOT started by this script (it never invokes those binaries), so unrelated to this run; investigate separately if unexpected"
|
||||
fi
|
||||
|
||||
local failed=0
|
||||
for pair in "${pairs[@]}"; do
|
||||
name="${pair%%=*}"
|
||||
case "${INJECTED_RESULTS[$name]}" in
|
||||
"rejected: no result line observed")
|
||||
log "FAIL: '$name' produced no observable accept/reject outcome -- structural injection failure, not a legitimate rejection"
|
||||
failed=1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
[ "$failed" -eq 0 ] || fail "injection did not produce a definitive accept/reject outcome for every requested resource (see above)"
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
resolve_top
|
||||
|
|
@ -284,20 +453,18 @@ main() {
|
|||
start_all
|
||||
confirm_deployment_up
|
||||
|
||||
# --- Extension point (task 6.1): join a synthetic worker via xce_test's
|
||||
# worker context and run resupdate for RESOURCE_ARGS (or the default
|
||||
# gap-resource set if RESOURCE_ARGS is empty) here. ---
|
||||
inject_resources "${RESOURCE_ARGS[@]}"
|
||||
|
||||
# --- Extension point (tasks 6.2-6.4): controller/scheduler/apiserver
|
||||
# per-layer verification checks here. ---
|
||||
# per-layer verification checks here, consuming
|
||||
# ${INJECTED_VALUES[@]}/${INJECTED_RESULTS[@]} and $WORKER_HOST
|
||||
# populated by inject_resources() above (also mirrored to
|
||||
# $TOP/logs/injection-results.tsv). ---
|
||||
# --- Extension point (task 7.1): aggregate results into a VERDICT line
|
||||
# plus the terminal PASS/FAIL oracle line, replacing this task's
|
||||
# placeholder success message and exit code below. ---
|
||||
|
||||
if [ "${#RESOURCE_ARGS[@]}" -gt 0 ]; then
|
||||
log "note: resource=value arguments (${RESOURCE_ARGS[*]}) were accepted but are not yet acted on -- injection lands in task 6.1"
|
||||
fi
|
||||
|
||||
log "PASS (task 5.1 scope): controller, scheduler, and apiserver confirmed up under $TOP; cleaning up"
|
||||
log "PASS (task 5.1+6.1 scope): controller, scheduler, and apiserver confirmed up; worker '$WORKER_HOST' joined and resupdate ran for ${#INJECTED_RESULTS[@]} resource(s) under $TOP (see $TOP/logs/injection-results.tsv for the accept/reject detail per name -- a rejection there is a captured outcome, not a script failure at this stage); per-layer verification and the overall PASS/FAIL verdict land in tasks 6.2-7.1; cleaning up"
|
||||
exit 0
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue