Rongsong Shen
1364664769
aggregate_verdict()/build_verdict_record() combine the three per-layer check results into a single per-resource comparison, self-contained and invokable with synthetic records (task 8.2). Uses ASCII Unit Separator (\x1f) as the record delimiter after discovering bash's IFS tab-splitting silently collapses adjacent empty fields even with IFS set to only a tab (bug-1581), which corrupted UNREACHABLE-record field alignment. Emits one machine-parseable VERDICT line (resources/layers_checked/ mismatches/unreachable/rejected/result) plus the terminal PASS/FAIL oracle line and matching exit code. Cleanup is the same trap-based mechanism task 5.1 registered for the abort path - logs always preserved regardless of --keep-running. Round-2 fix: main()'s live-data wiring (build_verdict_records_from_tsv()) derived a layer's status from array-key existence rather than the actual recorded outcome - since tasks 6.3/6.4 always set the key (even on a genuine timeout, just with an empty value), every real UNREACHABLE was silently misclassified as MISMATCH. Fixed to check SCHEDULER_CHECK_RESULTS/ APISERVER_CHECK_RESULTS string content instead (bug-1582). Live-confirmed by killing the scheduler mid-run: VERDICT now correctly reports unreachable=9 mismatches=0, not the reverse. SKILL.md rewritten to describe the now-complete, functional tool. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1551 lines
79 KiB
Bash
Executable file
1551 lines
79 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
#
|
|
# run-resource-verify.sh -- driver for /xce-test-resources: end-to-end
|
|
# resource-value propagation verification (inject a synthetic worker
|
|
# resource value via xce_test's resupdate, confirm it is visible at the
|
|
# controller, live scheduler, and Go apiserver).
|
|
#
|
|
# resource-value-verification spec, task 5.1+6.1+6.2+6.3+6.4+7.1 SCOPE (+
|
|
# bug-1579 follow-up fix): this file implements the FULL skill -- deployment
|
|
# (task 5.1), worker join + resource injection (task 6.1), the
|
|
# controller-side per-layer verification check (task 6.2), the
|
|
# scheduler-side per-layer verification check (task 6.3), the apiserver-side
|
|
# per-layer verification check (task 6.4), AND the final verdict aggregation
|
|
# / VERDICT line / PASS-FAIL oracle / cleanup-on-every-path (task 7.1) --
|
|
# stand up a fresh controller + scheduler
|
|
# (-vvv) + apiserver (debug logging) deployment via xce-deploy-ha's
|
|
# deploy-ha.sh + generated start-all.sh, confirm all three components are
|
|
# actually up, join a SYNTHETIC worker (no real worker daemon process)
|
|
# through xce_test's worker context, run resupdate for one or more
|
|
# resource=value pairs, then (a) query the controller (xce_test
|
|
# client-context `resval`) for every ACCEPTED pair and report
|
|
# MATCH/MISMATCH against the injected value, (b) poll the live scheduler
|
|
# daemon's own log file (task 4.1's trace line) for the same ACCEPTED pairs,
|
|
# within a bounded wait, and report MATCH/MISMATCH/UNREACHABLE distinctly
|
|
# from the controller-side result, and (c) poll the live apiserver's own log
|
|
# file (task 4.2's trace line) for the same ACCEPTED pairs, within a bounded
|
|
# wait, and report MATCH/MISMATCH/UNREACHABLE distinctly from both the
|
|
# controller-side and scheduler-side results, then clean up (trap-based,
|
|
# covering a mid-deployment abort, not just a completed run).
|
|
#
|
|
# bug-1579 follow-up fix (this revision): the controller-side check (a) above
|
|
# now runs NESTED inside inject_resources()'s own xce_test worker session --
|
|
# via a `!` shell-escape line placed between `resupdate` and `quit` -- rather
|
|
# than as a wholly separate xce_test process run AFTER that session's own
|
|
# `quit` has already disconnected the synthetic worker. See the
|
|
# internal_controller_check() function and the top-of-file bug-1579 note
|
|
# below for why this matters and exactly how it works; DEFAULT_RESOURCE_ARGS
|
|
# now includes `cpuf` again as proof the fix is real (see that note).
|
|
# See .kiro/specs/resource-value-verification/{requirements,design}.md,
|
|
# Requirements 2.1, 2.2, 3.1, 3.2, 4.1, 4.2, 4.3, 5.3, 6.1, 6.2, 6.3, 6.4,
|
|
# 6.5 for these tasks; the full skill covers 3.1-6.5 and is now complete.
|
|
#
|
|
# Task 7.1 (this revision): adds the final aggregation step. Combines the
|
|
# controller/scheduler/apiserver per-layer results (tasks 6.2-6.4) plus the
|
|
# injection accept/reject outcome (tasks 2.1/2.2/6.1) into a single overall
|
|
# verdict, via a SELF-CONTAINED comparison/aggregation function
|
|
# (aggregate_verdict(), see its own header comment below) that takes plain
|
|
# TSV "records" as arguments -- no live command, no file I/O, no global
|
|
# deployment state -- so it is directly callable with hand-written synthetic
|
|
# records for testing (this task's own sanity-check invocations, and task
|
|
# 8.2's dedicated harness, which depends on this exact function/record
|
|
# format) as well as from main()'s live-run path. Emits exactly one
|
|
# machine-parseable VERDICT line (xce-test-ha's convention) plus the
|
|
# standard terminal PASS/FAIL oracle line; exit code is 0 for an overall
|
|
# PASS, 1 for an overall FAIL -- this is now the script's REAL exit code,
|
|
# not the deployment-only placeholder described in the "Exit 0/Non-zero"
|
|
# note below the Usage block (kept for historical/task-5.1-6.4 context, see
|
|
# the correction directly beneath it). The bottom of this file is also now
|
|
# guarded so it can be `source`d (e.g. by task 8.2's harness) without
|
|
# executing main()/internal_controller_check() as a side effect -- see the
|
|
# `[[ "${BASH_SOURCE[0]}" == "${0}" ]]` guard at the very end.
|
|
#
|
|
# Task 6.1 deliberate deviation from design.md: the default (no-args)
|
|
# resource set is Requirement 1's gap-resource set (r15m, pg, ls, it, swp,
|
|
# io, tmp) plus a couple of already-supported NUMERIC names for regression
|
|
# coverage (mem, cpuf) -- NOT "type"/"model" as design.md's illustrative text
|
|
# suggests. bug-1574 (xce_test can SEGFAULT, non-deterministically, on
|
|
# `resupdate -n type/-n model ...` -- uninitialized-memory UB in
|
|
# set_resval()'s Value_u_svalue fallback path) makes including a string-typed
|
|
# resource in this script's own default live-testing path a real crash risk,
|
|
# not a theoretical one. Revisit once bug-1574 is fixed.
|
|
#
|
|
# Task 6.2 remediation (bug-1579, original workaround): the
|
|
# regression-coverage name was originally "cpuf", but cpuf is one of
|
|
# worker.c's N_HOST_ATTRS (add_hostattrs_resvals()) -- the controller RESETS
|
|
# it to baseline the moment the synthetic worker disconnects
|
|
# (mark_worker_unavail() -> apply_worker_dead_mutation() ->
|
|
# reset_hostattrs_unknown() + add_hostattrs_resvals()), and
|
|
# inject_resources()'s xce_test session used to end with `quit` (disconnecting
|
|
# the worker) before a separate, later xce_test process ran the controller
|
|
# query -- so cpuf always reported a false MISMATCH (observed=0.000000).
|
|
# Task 6.2's review swapped it for "r1m" (confirmed via
|
|
# resval_typeresolve.c's resinfo[] table to be a genuinely pre-existing
|
|
# ("already-supported") resource name, and via
|
|
# src/worker/src/resource_sync.c to be a periodic/dynamic load-index
|
|
# resource, not in worker.c's N_HOST_ATTRS list) as a workaround, not a fix --
|
|
# the underlying disconnect-timing problem remained live for any *other*
|
|
# host-attrs name (including a user-supplied custom resource=value
|
|
# argument).
|
|
#
|
|
# bug-1579 REAL FIX (this revision): xce_test's `!<shell-command>` escape
|
|
# (preprocess_and_run(), test.c ~line 950-998) is handled in the single
|
|
# top-level REPL loop (test.c main(), the sole call site) BEFORE any
|
|
# context-specific command dispatch -- so it works identically in ANY
|
|
# context, including `worker`, WITHOUT calling leave_ctx()/disconnecting.
|
|
# inject_resources() now inserts a `!` line between `resupdate` and `quit`
|
|
# that invokes this very script recursively, in a hidden
|
|
# `--internal-controller-check` mode (see internal_controller_check() below),
|
|
# from INSIDE the still-open worker session. That nested invocation runs its
|
|
# own client-context xce_test session and queries `resval` for every
|
|
# requested resource name WHILE the synthetic worker is still connected --
|
|
# so a host-attrs resource is never reset before it's checked. The nested
|
|
# check writes raw observed values to $TOP/logs/controller-check-raw.tsv;
|
|
# query_controller_resvals() (task 6.2's original function, still called at
|
|
# the same point in main(), after inject_resources() returns) now reads that
|
|
# file instead of running its own live query -- a SEQUENCING change, not a
|
|
# logic change: the MATCH/MISMATCH comparison, CONTROLLER_OBSERVED/
|
|
# CONTROLLER_CHECK_RESULTS population, and controller-check-results.tsv
|
|
# format are all unchanged. DEFAULT_RESOURCE_ARGS now restores "cpuf"
|
|
# alongside "mem" (design.md's original "a couple of already-supported
|
|
# names for regression coverage" intent) -- this is the proof the fix is
|
|
# real: cpuf now correctly MATCHes at the controller layer instead of
|
|
# false-MISMATCHing. The alternative considered during task 6.2's review
|
|
# (keep the worker session connected across all per-layer checks by
|
|
# reordering calls) was already ruled out there as not viable without this
|
|
# `!`-escape mechanism -- xce_test's REPL has no command path from the
|
|
# `worker` context back to `client` without leave_ctx() (`..`/cmd_up), which
|
|
# disconnects exactly like `quit` does. See bug-1579 for the full history.
|
|
#
|
|
# Must be invoked under an environment with `nix` on PATH (same as
|
|
# xce-deploy-ha / xce-test-ha) -- this script shells out to `nix develop`
|
|
# itself for the steps that need it (openssl, xceadmin), so it does not need
|
|
# to be pre-wrapped by the caller.
|
|
#
|
|
# Usage:
|
|
# run-resource-verify.sh [--keep-running] [--dir DIR] [resource=value ...]
|
|
#
|
|
# --keep-running Skip PROCESS teardown in cleanup -- the deployment
|
|
# (controller/scheduler/apiserver) is left running
|
|
# under DIR for further inspection. Log files under
|
|
# DIR are ALWAYS preserved regardless of this flag --
|
|
# it only controls whether processes are killed (see
|
|
# cleanup(), and design.md's explicit divergence from
|
|
# xce-deploy-ha's own cleanup.sh, which wipes logs).
|
|
# --dir DIR Scratch/deploy directory. Default: a fresh directory
|
|
# generated via `mktemp -d`, following xce-deploy-ha's
|
|
# "every invocation is a fresh, self-contained deploy"
|
|
# convention. $XCE_RESVERIFY_TOP overrides the default
|
|
# when --dir is not given (same precedence pattern as
|
|
# xce-test-ha's $XCE_HA_TOP).
|
|
# resource=value One or more resources to inject via resupdate, e.g.
|
|
# `r15m=1.5 pg=3`. May repeat. When none are given,
|
|
# defaults to Requirement 1's gap-resource set plus a
|
|
# couple of already-numeric names for regression
|
|
# coverage (see the top-of-file bug-1574 note for why
|
|
# the default deliberately excludes type/model).
|
|
#
|
|
# --- SUPERSEDED by task 7.1 (kept for historical context -- this described
|
|
# the script's exit code BEFORE the final aggregation step existed) ---
|
|
# Exit 0 == this task's scope succeeded (deployment stood up, all three
|
|
# components confirmed up, worker joined, resupdate ran for every requested
|
|
# resource -- accepted AND rejected outcomes are both a successful run of
|
|
# THIS script's scope, see inject_resources() -- and the controller-side
|
|
# check (task 6.2), scheduler-side check (task 6.3), and apiserver-side check
|
|
# (task 6.4) each ran for every accepted resource -- a MATCH, a MISMATCH, or
|
|
# (scheduler-side/apiserver-side only) an UNREACHABLE/timeout are ALL a
|
|
# successful run of this script's own scope, see
|
|
# query_controller_resvals()/query_scheduler_resvals()/
|
|
# query_apiserver_resvals(); only a structural failure to query the
|
|
# controller, or to read the injection manifest for the scheduler/apiserver
|
|
# checks, is fatal here -- cleaned up per --keep-running).
|
|
# Non-zero == deployment failed at some step, the worker failed to join,
|
|
# resupdate produced no observable accept/reject outcome for some name (a
|
|
# structural injection failure, distinct from a legitimate rejection), or the
|
|
# controller-side check itself could not run (e.g. the client context failed
|
|
# to connect). Once task 7.1 lands, the exit code instead reflects the full
|
|
# injection/verification PASS/FAIL verdict -- tasks 6.2/6.3/6.4 deliberately
|
|
# do NOT fail the run on a controller-, scheduler-, or apiserver-layer
|
|
# MISMATCH/UNREACHABLE yet (see ANY_CONTROLLER_MISMATCH/
|
|
# ANY_SCHEDULER_MISMATCH/ANY_SCHEDULER_UNREACHABLE/ANY_APISERVER_MISMATCH/
|
|
# ANY_APISERVER_UNREACHABLE, groundwork for task 7.1's aggregation).
|
|
# --- CURRENT (task 7.1, this revision) ---
|
|
# Exit 0 == overall verdict PASS: every requested resource name was accepted
|
|
# (Requirement 1.2's rejection path never fired) AND every accepted
|
|
# resource's controller/scheduler/apiserver observed value matched its
|
|
# injected value (see aggregate_verdict() below) -- exactly what the
|
|
# script's single VERDICT line's `result=PASS` reports.
|
|
# Non-zero (1) == overall verdict FAIL: at least one requested resource name
|
|
# was rejected, OR at least one accepted resource's value did not match (or
|
|
# was UNREACHABLE/timed out) at some layer -- exactly what `result=FAIL`
|
|
# reports, with every specific (resource, layer) failure named in the
|
|
# FAILURE lines aggregate_verdict() logs immediately above the VERDICT line.
|
|
# A structural failure before aggregation is reached (deployment failure,
|
|
# worker-join failure, a resupdate with no observable accept/reject outcome,
|
|
# a malformed `--dir`/argument, etc.) still exits non-zero via fail()/usage()
|
|
# (1, or 2 for a usage/argument error) -- unchanged from before this task.
|
|
#
|
|
set -u
|
|
|
|
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO="$(cd "$SELF_DIR/../../.." && pwd)"
|
|
# Absolute path to this script itself -- used by inject_resources()'s `!`
|
|
# shell-escape (bug-1579 fix, see the top-of-file note) to invoke this same
|
|
# script recursively in `--internal-controller-check` mode, from INSIDE the
|
|
# xce_test worker session's `system()` call, while that session's connection
|
|
# to the controller is still open (before its `quit`).
|
|
SELF_SCRIPT="$SELF_DIR/$(basename "${BASH_SOURCE[0]}")"
|
|
|
|
DEPLOY_HA="$REPO/src/tools/deploy-ha.sh"
|
|
BUILD_DIR="${XCE_RESVERIFY_BUILD_DIR:-$REPO/src/xce.build}"
|
|
BINARY="${XCE_RESVERIFY_BIN:-$BUILD_DIR/controller/xce_controller}"
|
|
SCHED_BIN="${XCE_RESVERIFY_SCHED_BIN:-$BUILD_DIR/scheduler/xce_scheduler}"
|
|
APISERVER_BIN="${XCE_RESVERIFY_APISERVER_BIN:-/tmp/apiserver}"
|
|
# xce_test -- drives injection (task 6.1) via its worker-context `join`/
|
|
# `resupdate` commands (same binary xce-test-ha's run-ha-soak.sh uses, same
|
|
# build dir as $BINARY since both come from the controller meson target).
|
|
XT="${XCE_RESVERIFY_XT:-$BUILD_DIR/controller/xce_test}"
|
|
|
|
# Single-node cluster, deliberately -- this skill verifies resource-value
|
|
# PROPAGATION (controller -> scheduler -> apiserver), not raft/HA failover
|
|
# behavior (that's xce-test-ha's job). A single node still elects itself
|
|
# CTRL_PRIMARY immediately and satisfies deploy-ha.sh's health-check, with
|
|
# no election-timing flakiness to budget for. Override for multi-node via
|
|
# $XCE_RESVERIFY_PEERS (comma-separated host:port list).
|
|
PEERS="${XCE_RESVERIFY_PEERS:-localhost:9991}"
|
|
|
|
KEEP_RUNNING=0
|
|
TOP=""
|
|
RESOURCE_ARGS=() # task 6.1 extension point: resource=value pairs
|
|
|
|
log() { echo "[xce-test-resources] $*"; }
|
|
fail() { echo "[xce-test-resources] FAIL: $*" >&2; exit 1; }
|
|
|
|
usage() {
|
|
cat <<EOF
|
|
Usage: $0 [--keep-running] [--dir DIR] [resource=value ...]
|
|
|
|
--keep-running leave controller/scheduler/apiserver processes running
|
|
after this run instead of tearing them down. Logs are
|
|
always preserved either way.
|
|
--dir DIR scratch/deploy directory (default: fresh mktemp -d, or
|
|
\$XCE_RESVERIFY_TOP).
|
|
resource=value inject this resource via resupdate (may repeat). Default
|
|
when none given: r15m=1.5 pg=2.5 ls=3.5 it=4.5 swp=5.5
|
|
io=6.5 tmp=7.5 mem=8.5 cpuf=9.5 (numeric-only -- see
|
|
bug-1574 note at top of file).
|
|
EOF
|
|
}
|
|
|
|
parse_args() {
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--keep-running)
|
|
KEEP_RUNNING=1
|
|
shift
|
|
;;
|
|
--dir)
|
|
[ $# -ge 2 ] || fail "--dir requires an argument"
|
|
TOP="$2"
|
|
shift 2
|
|
;;
|
|
--dir=*)
|
|
TOP="${1#--dir=}"
|
|
shift
|
|
;;
|
|
-h|--help)
|
|
usage
|
|
exit 0
|
|
;;
|
|
*=*)
|
|
# resource=value pair -- task 6.1's concern; just collected here.
|
|
RESOURCE_ARGS+=("$1")
|
|
shift
|
|
;;
|
|
*)
|
|
echo "[xce-test-resources] Unknown argument: $1" >&2
|
|
usage
|
|
exit 2
|
|
;;
|
|
esac
|
|
done
|
|
}
|
|
|
|
resolve_top() {
|
|
if [ -z "$TOP" ]; then
|
|
TOP="${XCE_RESVERIFY_TOP:-$(mktemp -d /tmp/xce-resverify.XXXXXX)}"
|
|
fi
|
|
# Resolve a relative --dir/$XCE_RESVERIFY_TOP against cwd, so it is
|
|
# unambiguous to deploy-ha.sh and every generated per-component script
|
|
# (same pattern as xce-test-ha's run-ha-soak.sh; no realpath dependency
|
|
# since busybox realpath in this environment lacks -m).
|
|
case "$TOP" in
|
|
/*) : ;;
|
|
*) TOP="$PWD/$TOP" ;;
|
|
esac
|
|
mkdir -p "$TOP"
|
|
}
|
|
|
|
# ── Cleanup: process teardown only, logs always preserved ──────────────────
|
|
# Deliberately does NOT invoke xce-deploy-ha's own generated cleanup.sh --
|
|
# that script wipes $TOP/logs/*.log, which this feature's design explicitly
|
|
# diverges from (a FAIL verdict's underlying scheduler/apiserver trace-log
|
|
# evidence must stay inspectable after the run). Every pkill pattern below is
|
|
# scoped to a $TOP-unique argv substring, not a bare binary/process name --
|
|
# xce-test-ha's bug-1436 postmortem: an unscoped `pkill -9 -x xce_controller`
|
|
# can kill a DIFFERENT concurrent run's live cluster sharing the same
|
|
# machine. $TOP itself (mktemp-unique per invocation, or an operator-chosen
|
|
# --dir) is what makes each of these patterns safe to run concurrently with
|
|
# an unrelated xce-deploy-ha/xce-test-ha/xce-test-resources invocation.
|
|
cleanup() {
|
|
if [ "$KEEP_RUNNING" -eq 1 ]; then
|
|
log "--keep-running requested -- leaving controller/scheduler/apiserver up under $TOP (logs also preserved)"
|
|
return 0
|
|
fi
|
|
log "cleaning up processes under $TOP (log files under $TOP/logs are preserved)"
|
|
# One combined pattern (alternation), not four separate pkill
|
|
# invocations -- fewer fork/exec round trips for a cleanup path that
|
|
# may run from a signal trap, plus retry-until-verified below so a
|
|
# slow/racy single kill attempt can't leave a process behind silently.
|
|
local pattern="--workdir $TOP/controller| -l $TOP/logs|--logfile $TOP/logs/scheduler.log| -d $TOP/apiserver"
|
|
local attempt=0
|
|
while [ "$attempt" -lt 10 ]; do
|
|
attempt=$((attempt + 1))
|
|
if ! pgrep -f -- "$pattern" >/dev/null 2>&1; then
|
|
break
|
|
fi
|
|
pkill -9 -f -- "$pattern" 2>/dev/null
|
|
sleep 0.2
|
|
done
|
|
if pgrep -f -- "$pattern" >/dev/null 2>&1; then
|
|
log "warning: some processes under $TOP may still be running after cleanup retries"
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# Standard robust trap idiom: on_exit (registered for EXIT) always runs
|
|
# cleanup and preserves whatever exit code the script was already leaving
|
|
# with. INT/TERM are trapped SEPARATELY and translated into an explicit
|
|
# `exit`, because a bare `trap cleanup INT TERM` (with no exit inside the
|
|
# handler) would only interrupt whatever foreground command was running and
|
|
# then RESUME the script at the next statement -- silently defeating a
|
|
# "kill the script mid-deployment" abort instead of actually stopping it.
|
|
# Chaining through `exit N` here re-triggers the EXIT trap, so cleanup only
|
|
# ever runs once.
|
|
on_exit() {
|
|
local rc=$?
|
|
cleanup
|
|
exit "$rc"
|
|
}
|
|
|
|
register_trap() {
|
|
trap on_exit EXIT
|
|
trap 'exit 130' INT
|
|
trap 'exit 143' TERM
|
|
}
|
|
|
|
precondition_checks() {
|
|
[ -x "$BINARY" ] || fail "controller binary missing: $BINARY (run /xce-build controller)"
|
|
[ -x "$SCHED_BIN" ] || fail "scheduler binary missing: $SCHED_BIN (run /xce-build scheduler) -- required, this skill verifies the scheduler layer"
|
|
[ -x "$APISERVER_BIN" ] || fail "apiserver binary missing: $APISERVER_BIN (cd src/frontend && make build-server, or the CGO-flags workaround -- see tasks.md's Implementation Notes) -- required, this skill verifies the apiserver layer"
|
|
[ -x "$XT" ] || fail "xce_test binary missing: $XT (run /xce-build controller) -- required to join a synthetic worker and run resupdate"
|
|
command -v nix >/dev/null 2>&1 || fail "nix not found on PATH -- required to resolve the deploy-environment bash and invoke deploy-ha.sh"
|
|
}
|
|
|
|
# ── Step 1: deploy (xce-deploy-ha's deploy-ha.sh) ───────────────────────────
|
|
deploy() {
|
|
log "resolving deploy-environment bash via nix develop"
|
|
BASH_BIN="$(nix develop "$REPO/src" --command which bash)"
|
|
[ -n "$BASH_BIN" ] || fail "deployment failed at step deploy-ha.sh: could not resolve bash inside nix develop"
|
|
|
|
log "deploying controller+scheduler+apiserver into $TOP (peers=$PEERS)"
|
|
if ! nix develop "$REPO/src" --command "$BASH_BIN" "$DEPLOY_HA" "$BINARY" "$TOP" "$PEERS"; then
|
|
fail "deployment failed at step deploy-ha.sh (see output above)"
|
|
fi
|
|
|
|
[ -x "$TOP/scripts/start-all.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start-all.sh was not generated"
|
|
[ -x "$TOP/scripts/start-scheduler.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start-scheduler.sh was not generated (scheduler binary not found alongside controller build?)"
|
|
[ -x "$TOP/scripts/start-apiserver.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start-apiserver.sh was not generated (no apiserver binary at $APISERVER_BIN?)"
|
|
}
|
|
|
|
# ── Step 2: start-all.sh -- controller -> health-check -> scheduler/apiserver
|
|
start_all() {
|
|
# deploy-ha.sh's Step 3 also generates + start-all.sh also launches a
|
|
# REAL worker daemon (xcecd_virtual) whenever one happens to exist
|
|
# alongside $BINARY in the build dir (its own pre-existing, opt-in-by-
|
|
# presence worker/scheduler convention, predating this feature -- see
|
|
# task 3.1's notes). This skill's whole point (Requirement 2.1) is that
|
|
# NO real worker daemon process is needed -- task 6.1 injects via a
|
|
# SYNTHETIC xce_test-joined worker instead -- and a real worker sitting
|
|
# in the background for this run's whole duration is worse than just
|
|
# off-contract: it periodically re-reports its OWN resources (~every
|
|
# 15s), and a resource's SECOND periodic report is bug-1572 (open,
|
|
# unfixed scheduler double-free/crash) -- a real background worker is a
|
|
# live crash risk to this run, not just noise. start-worker.sh reads
|
|
# $XCE_WORKER_BIN itself (falling back to the real xcecd_virtual path)
|
|
# -- point it at a real no-op binary so start-all.sh's worker step exits
|
|
# immediately instead of leaving a daemon running; deploy-ha.sh itself
|
|
# is untouched (out of boundary), this only affects what THIS skill's
|
|
# own start-all.sh invocation actually launches.
|
|
export XCE_WORKER_BIN="/bin/true"
|
|
|
|
log "invoking start-all.sh (controller, scheduler at -vvv, apiserver with debug logging; real worker daemon suppressed via XCE_WORKER_BIN=/bin/true -- see comment above)"
|
|
if ! "$BASH_BIN" "$TOP/scripts/start-all.sh"; then
|
|
fail "deployment failed at step start-all.sh (nonzero exit)"
|
|
fi
|
|
}
|
|
|
|
# ── Step 3: confirm all three components are actually up ───────────────────
|
|
# start-all.sh's own internal health-check only gates on the CONTROLLER
|
|
# electing a primary before it goes on to background scheduler/apiserver --
|
|
# it does not itself wait for the scheduler to finish joining the cluster or
|
|
# for the apiserver to finish registering with the controller before
|
|
# returning. This skill's own "deployment succeeded" signal (distinct from
|
|
# start-all.sh's) requires all three, so poll each with a short interval up
|
|
# to a generous bounded wait (this codebase's established convention for
|
|
# this kind of check -- see xce-test-ha and design.md's Implementation
|
|
# Notes for the xce-test-resources skill).
|
|
wait_for_log() {
|
|
local desc="$1" pattern="$2" timeout="$3"
|
|
shift 3
|
|
local waited=0
|
|
while [ "$waited" -lt "$timeout" ]; do
|
|
if grep -qE -- "$pattern" "$@" 2>/dev/null; then
|
|
log "confirmed: $desc"
|
|
return 0
|
|
fi
|
|
sleep 1
|
|
waited=$((waited + 1))
|
|
done
|
|
return 1
|
|
}
|
|
|
|
confirm_deployment_up() {
|
|
log "waiting for controller, scheduler, and apiserver to confirm up (bounded wait per component)"
|
|
|
|
wait_for_log "controller primary elected" "CTRL_PRIMARY" 30 $TOP/logs/ctrl-*.log \
|
|
|| fail "deployment failed at step controller: no primary election observed in $TOP/logs/ctrl-*.log within 30s"
|
|
|
|
wait_for_log "scheduler joined cluster" "sched_init: joined cluster" 30 "$TOP/logs/scheduler.log" \
|
|
|| fail "deployment failed at step scheduler: '$TOP/logs/scheduler.log' never logged joining the cluster within 30s"
|
|
|
|
wait_for_log "apiserver backend connection registered" "Backend connection registered" 30 "$TOP/logs/apiserver.log" \
|
|
|| fail "deployment failed at step apiserver: '$TOP/logs/apiserver.log' never logged a registered backend connection within 30s"
|
|
|
|
log "all three components confirmed up under $TOP"
|
|
}
|
|
|
|
# ── Step 4: join a synthetic worker + inject resource values (task 6.1) ────
|
|
# Requirement 1's gap-resource set (numeric) plus a couple of already-numeric
|
|
# names for regression coverage -- see the top-of-file bug-1574 note for why
|
|
# this deliberately excludes type/model (design.md's own illustrative text
|
|
# names them, but xce_test can SEGFAULT on a string-typed resupdate, so this
|
|
# is a documented, deliberate deviation, not an oversight). The two
|
|
# regression-coverage names are "mem" and "cpuf", matching design.md's
|
|
# original "a couple of already-supported names" intent -- "cpuf" is a
|
|
# worker.c N_HOST_ATTRS name and is restored here (bug-1579's real fix, see
|
|
# the top-of-file note) as live proof that the controller-side check no
|
|
# longer false-MISMATCHes a host-attrs resource: the nested `!`-escape
|
|
# controller check (internal_controller_check()) runs while the synthetic
|
|
# worker's connection is still open, so cpuf is never reset to baseline
|
|
# before it's checked.
|
|
DEFAULT_RESOURCE_ARGS=(
|
|
r15m=1.5 pg=2.5 ls=3.5 it=4.5 swp=5.5 io=6.5 tmp=7.5
|
|
mem=8.5 cpuf=9.5
|
|
)
|
|
|
|
# Fixed synthetic worker identity for this run -- tags every resupdate'd
|
|
# value (Requirement 2.2), exactly like `set myname <host>` + `worker` +
|
|
# `join` in xce-test-ha's run-ha-soak.sh (identical established pattern).
|
|
# A fresh $TOP per invocation means no cross-run collision risk.
|
|
WORKER_HOST="resverify-worker"
|
|
|
|
declare -A INJECTED_VALUES=() # resource name -> injected value, as sent
|
|
declare -A INJECTED_RESULTS=() # resource name -> "accepted" | "rejected: <reason>" | "rejected: no result line observed"
|
|
|
|
# Run a batch of xce_test commands (stdin script). Two things this MUST do,
|
|
# both hard-won (see cerebrum.md / buglog.json):
|
|
# - ASAN_OPTIONS=detect_leaks=0: xce_test is an ASAN build with small
|
|
# pre-existing intentional REPL-lifetime leaks; its default leak-detect
|
|
# exit path can swallow buffered stdout on a piped/redirected run,
|
|
# losing the very accept/reject lines this function parses.
|
|
# - wrapped in `nix develop` (like deploy()'s own BASH_BIN resolution
|
|
# above): unlike $BINARY/$SCHED_BIN/$APISERVER_BIN (started via
|
|
# generated scripts whose binaries carry a baked RPATH), xce_test's own
|
|
# runtime linkage has not been confirmed independent of the nix
|
|
# devShell's library path -- wrap defensively, mirroring deploy()'s own
|
|
# "shell out to nix develop for the steps that need it" posture.
|
|
xtest() {
|
|
nix develop "$REPO/src" --command env ASAN_OPTIONS=detect_leaks=0 timeout 60 "$XT" 2>&1
|
|
}
|
|
|
|
# inject_resources [resource=value ...]
|
|
# No args -> DEFAULT_RESOURCE_ARGS. Joins one synthetic worker (no real
|
|
# worker daemon process anywhere in this flow -- Requirement 2.1) and runs
|
|
# ONE resupdate invocation carrying every pair, so task 2.2's per-name
|
|
# accept/reject accounting covers the whole batch in a single RPC round
|
|
# trip. Populates INJECTED_VALUES/INJECTED_RESULTS (and $TOP/logs/
|
|
# injection-results.tsv) for tasks 6.2-6.4's extension points to consume.
|
|
inject_resources() {
|
|
local pairs=("$@")
|
|
local pair name value resupdate_line out result
|
|
|
|
if [ "${#pairs[@]}" -eq 0 ]; then
|
|
pairs=("${DEFAULT_RESOURCE_ARGS[@]}")
|
|
fi
|
|
|
|
resupdate_line=""
|
|
for pair in "${pairs[@]}"; do
|
|
name="${pair%%=*}"
|
|
value="${pair#*=}"
|
|
if [ -z "$name" ] || [ "$name" = "$pair" ] || [ -z "$value" ]; then
|
|
fail "malformed resource argument (expected resource=value): $pair"
|
|
fi
|
|
INJECTED_VALUES["$name"]="$value"
|
|
resupdate_line="$resupdate_line -n $name -v $value"
|
|
done
|
|
|
|
log "joining synthetic worker '$WORKER_HOST' (masters=$PEERS) and injecting: ${pairs[*]}"
|
|
|
|
# bug-1579 fix: run the controller-side check (internal_controller_check(),
|
|
# below) via a `!` shell-escape BEFORE `quit` -- while this session's
|
|
# worker connection is still open -- instead of in a separate xce_test
|
|
# process run after this session has already disconnected. `%q`-quote
|
|
# every interpolated value defensively (none of TOP/PEERS/WORKER_HOST/
|
|
# resource names are expected to need it, but this line is re-parsed by
|
|
# a shell via xce_test's system() call, so quote anyway).
|
|
local ctrl_check_cmd
|
|
ctrl_check_cmd="bash $(printf '%q' "$SELF_SCRIPT") --internal-controller-check $(printf '%q' "$TOP") $(printf '%q' "$PEERS") $(printf '%q' "$WORKER_HOST")"
|
|
for pair in "${pairs[@]}"; do
|
|
name="${pair%%=*}"
|
|
ctrl_check_cmd="$ctrl_check_cmd $(printf '%q' "$name")"
|
|
done
|
|
|
|
local script
|
|
script="$(printf 'set masters %s\nset myname %s\nworker\njoin\nresupdate%s\n!%s\nquit\n' \
|
|
"$PEERS" "$WORKER_HOST" "$resupdate_line" "$ctrl_check_cmd")"
|
|
|
|
out="$(printf '%s' "$script" | xtest)"
|
|
|
|
printf '%s\n' "$out" | grep -Fxq -- "host join ok" \
|
|
|| fail "worker join failed -- no synthetic worker daemon process exists so this is the whole injection path; xce_test output:
|
|
$out"
|
|
|
|
for pair in "${pairs[@]}"; do
|
|
name="${pair%%=*}"
|
|
if printf '%s\n' "$out" | grep -Fxq -- "${name}: accepted"; then
|
|
result="accepted"
|
|
elif printf '%s\n' "$out" | grep -Fxq -- "${name}: rejected: unknown resource"; then
|
|
result="rejected: unknown resource"
|
|
elif printf '%s\n' "$out" | grep -Fxq -- "${name}: rejected: value mismatch"; then
|
|
result="rejected: value mismatch"
|
|
else
|
|
result="rejected: no result line observed"
|
|
fi
|
|
INJECTED_RESULTS["$name"]="$result"
|
|
log "resupdate: $name=${INJECTED_VALUES[$name]} -> $result"
|
|
done
|
|
|
|
# Manifest for post-run inspection and for tasks 6.2-6.4 to consume
|
|
# (this same script process already has INJECTED_VALUES/INJECTED_RESULTS
|
|
# in memory; the file is a durable, human-inspectable mirror of the
|
|
# same data, alongside the always-preserved logs/ convention).
|
|
mkdir -p "$TOP/logs"
|
|
{
|
|
for pair in "${pairs[@]}"; do
|
|
name="${pair%%=*}"
|
|
printf '%s\t%s\t%s\n' "$name" "${INJECTED_VALUES[$name]}" "${INJECTED_RESULTS[$name]}"
|
|
done
|
|
} > "$TOP/logs/injection-results.tsv"
|
|
|
|
# This flow never starts a real worker daemon (xcecd/xcecd_virtual) --
|
|
# structurally guaranteed by construction (the only process this
|
|
# function starts is xce_test itself, and only its worker CONTEXT, not
|
|
# a separate daemon binary). Confirm explicitly rather than just assert
|
|
# it in a comment, since Requirement 2.1/6.4's whole point is "no real
|
|
# worker daemon process required".
|
|
if pgrep -f -- 'xcecd_virtual|/xcecd( |$)' >/dev/null 2>&1; then
|
|
log "note: a process matching xcecd/xcecd_virtual is running on this machine -- NOT started by this script (it never invokes those binaries), so unrelated to this run; investigate separately if unexpected"
|
|
fi
|
|
|
|
local failed=0
|
|
for pair in "${pairs[@]}"; do
|
|
name="${pair%%=*}"
|
|
case "${INJECTED_RESULTS[$name]}" in
|
|
"rejected: no result line observed")
|
|
log "FAIL: '$name' produced no observable accept/reject outcome -- structural injection failure, not a legitimate rejection"
|
|
failed=1
|
|
;;
|
|
esac
|
|
done
|
|
[ "$failed" -eq 0 ] || fail "injection did not produce a definitive accept/reject outcome for every requested resource (see above)"
|
|
}
|
|
|
|
# ── Step 5: controller-side verification check (task 6.2 + bug-1579 fix) ───
|
|
# Requirement 3.1/3.2: for every SUCCESSFULLY-injected (accepted) resource/
|
|
# worker pair, query the controller via xce_test's client-context `resval`
|
|
# (the same xcecli_resval() call cmd_resupdate's own task 2.2 follow-up
|
|
# verification already uses internally -- a READ query, not a mutation, low
|
|
# risk) and compare the returned value against what was injected. A REJECTED
|
|
# resource was never actually applied at the controller in the first place
|
|
# (task 2.2's own follow-up query inside resupdate already confirmed that at
|
|
# injection time) -- there is nothing to verify for it here, so
|
|
# query_controller_resvals() only REPORTS on names marked "accepted" in
|
|
# $TOP/logs/injection-results.tsv.
|
|
#
|
|
# bug-1579 fix: the live `resval` query itself no longer happens here. It now
|
|
# happens EARLIER, nested inside inject_resources()'s own xce_test worker
|
|
# session via a `!` shell-escape (see internal_controller_check() below and
|
|
# the top-of-file bug-1579 note) -- for EVERY requested name, accepted or
|
|
# not, since accept/reject isn't known yet at that point in the sequence.
|
|
# query_controller_resvals() reads that nested query's raw results back from
|
|
# $TOP/logs/controller-check-raw.tsv and reports only on the names
|
|
# $TOP/logs/injection-results.tsv (available only now, after
|
|
# inject_resources() has returned and parsed the whole session's output)
|
|
# marks "accepted" -- a pure sequencing change: the MATCH/MISMATCH
|
|
# comparison, CONTROLLER_OBSERVED/CONTROLLER_CHECK_RESULTS population, and
|
|
# controller-check-results.tsv format below are all unchanged from task 6.2.
|
|
|
|
# internal_controller_check TOP PEERS WORKER_HOST NAME...
|
|
# NOT a normal in-process function call -- invoked as a FRESH
|
|
# `bash $SELF_SCRIPT --internal-controller-check ...` process via
|
|
# inject_resources()'s `!` shell-escape line, from INSIDE the xce_test
|
|
# worker-context session's system() call, while the injecting synthetic
|
|
# worker's connection to the controller is STILL OPEN (i.e. before that same
|
|
# session's `quit`). This is what makes the controller-side check immune to
|
|
# the worker-disconnect host-attrs reset (worker.c's mark_worker_unavail()
|
|
# -> apply_worker_dead_mutation() -> reset_hostattrs_unknown() +
|
|
# add_hostattrs_resvals(), N_HOST_ATTRS) -- the worker never disconnects
|
|
# before this check runs.
|
|
#
|
|
# Queries resval for EVERY given name, not just ones that will turn out to be
|
|
# "accepted" -- cmd_resupdate()'s own accept/reject determination has already
|
|
# run inside THIS SAME xce_test process by this point and been printed to
|
|
# its stdout, but that text isn't available to this freshly-spawned process;
|
|
# accept/reject filtering happens later, in query_controller_resvals(), once
|
|
# the parent script has parsed the whole session's captured output. Writes
|
|
# RAW observed values (blank if not found) to
|
|
# $TOP/logs/controller-check-raw.tsv (name\tobserved) for
|
|
# query_controller_resvals() to read once inject_resources() returns.
|
|
#
|
|
# On a connection failure, records $TOP/logs/controller-check-error.txt
|
|
# instead (query_controller_resvals() fails the run on this, same failure
|
|
# semantics as task 6.2's original inline check) and deliberately still
|
|
# returns 0 either way: a non-zero exit here becomes this xce_test session's
|
|
# g_last_status and, in non-interactive/batch mode, ABORTS its REPL loop
|
|
# before `quit` runs (preprocess_and_run()'s shell-escape handling) --
|
|
# undesirable when the failure can instead be surfaced through the raw-file/
|
|
# error-file contract and let this session reach `quit` normally.
|
|
internal_controller_check() {
|
|
local top="$1" peers="$2" worker_host="$3"
|
|
shift 3
|
|
local names=("$@")
|
|
|
|
mkdir -p "$top/logs"
|
|
local raw_tsv="$top/logs/controller-check-raw.tsv"
|
|
local err_txt="$top/logs/controller-check-error.txt"
|
|
rm -f "$err_txt"
|
|
: > "$raw_tsv"
|
|
|
|
[ "${#names[@]}" -gt 0 ] || return 0
|
|
|
|
local cmds="" name
|
|
for name in "${names[@]}"; do
|
|
cmds="${cmds}resval -n $name -m $worker_host"$'\n'
|
|
done
|
|
|
|
local script out
|
|
script="$(printf 'set masters %s\nclient\n%squit\n' "$peers" "$cmds")"
|
|
out="$(printf '%s' "$script" | xtest)"
|
|
|
|
if printf '%s\n' "$out" | grep -Fq -- "enter_ctx: invalid"; then
|
|
printf '%s\n' "$out" > "$err_txt"
|
|
return 0
|
|
fi
|
|
|
|
# Same sequential state-machine parse query_controller_resvals() used to
|
|
# do inline pre-fix -- see its own historical comment (still applicable
|
|
# here) for why (bug-1576's short-name substring hazard, no trailing
|
|
# marker after the last query).
|
|
local -A parsed_values=()
|
|
local line cur_name=""
|
|
while IFS= read -r line; do
|
|
case "$line" in
|
|
"resource "*"@${worker_host}:")
|
|
cur_name="${line#resource }"
|
|
cur_name="${cur_name%@${worker_host}:}"
|
|
;;
|
|
"value: "*)
|
|
if [ -n "$cur_name" ]; then
|
|
parsed_values["$cur_name"]="${line#value: }"
|
|
fi
|
|
cur_name=""
|
|
;;
|
|
esac
|
|
done <<< "$out"
|
|
|
|
for name in "${names[@]}"; do
|
|
printf '%s\t%s\n' "$name" "${parsed_values[$name]:-}" >> "$raw_tsv"
|
|
done
|
|
return 0
|
|
}
|
|
|
|
declare -A CONTROLLER_OBSERVED=() # resource name -> raw value string as rendered by cmd_resval (only set for accepted names actually queried)
|
|
declare -A CONTROLLER_CHECK_RESULTS=() # resource name -> "MATCH" | "MISMATCH: injected=<X> observed=<Y>" | "MISMATCH: controller returned no value ..."
|
|
ANY_CONTROLLER_MISMATCH=0 # groundwork for task 7.1's verdict aggregation; this task's own exit code does NOT depend on this -- see the top-of-file Exit code note and main()
|
|
|
|
# controller_values_match INJECTED OBSERVED
|
|
# Pure comparison helper (Requirement 3.2 / design.md's Error Handling "Layer
|
|
# mismatch" convention: report both values on a mismatch). Deliberately
|
|
# factored out of query_controller_resvals() so it can be exercised directly
|
|
# with synthetic inputs, independent of any live controller query -- this
|
|
# task's validation approach exercises exactly this (see SKILL.md / task
|
|
# notes). Returns 0 (bash "true"/success) on a match, 1 on a mismatch.
|
|
#
|
|
# Numeric comparison, not string equality: cmd_resval's own controller-side
|
|
# rendering always prints fixed 6-decimal %f notation (e.g. an injected
|
|
# "1.5" comes back rendered as "1.500000"; a bytes-valued resource such as
|
|
# `mem`/`tmp` additionally appends a human-readable "(X.XX UNIT)" suffix
|
|
# after the raw float, already stripped by the caller before this function
|
|
# ever sees it) -- a literal string compare would falsely MISMATCH on every
|
|
# numeric resource. Mirrors task 4.1's identical scheduler trace-log
|
|
# parse-as-float lesson (see tasks.md's Implementation Notes). Falls back to
|
|
# exact string equality for non-numeric (string-typed, e.g. type/model)
|
|
# values -- this script's own default resource set excludes those
|
|
# (bug-1574), but the comparison helper itself stays correct for either
|
|
# kind, since a future caller (or a synthetic test invocation) may pass one.
|
|
controller_values_match() {
|
|
local injected="$1" observed="$2"
|
|
if [[ "$injected" =~ ^-?[0-9]+([.][0-9]+)?$ ]] && [[ "$observed" =~ ^-?[0-9]+([.][0-9]+)?$ ]]; then
|
|
awk -v a="$injected" -v b="$observed" \
|
|
'BEGIN { d = a - b; if (d < 0) d = -d; exit !(d < 0.0005) }'
|
|
else
|
|
[ "$injected" = "$observed" ]
|
|
fi
|
|
}
|
|
|
|
# query_controller_resvals -- reads $TOP/logs/injection-results.tsv (task
|
|
# 6.1's durable, order-preserving mirror of INJECTED_VALUES/INJECTED_RESULTS
|
|
# -- bash associative arrays have no defined iteration order, the TSV does)
|
|
# and, for every name marked "accepted", issues one `resval -n <name> -m
|
|
# <host>` query in the xce_test client context, batched into a single
|
|
# xtest() invocation (same "one script, one process" precedent as
|
|
# inject_resources()'s single combined resupdate call). Populates
|
|
# CONTROLLER_OBSERVED/CONTROLLER_CHECK_RESULTS and
|
|
# $TOP/logs/controller-check-results.tsv (name\tinjected\tobserved\tresult).
|
|
query_controller_resvals() {
|
|
local tsv="$TOP/logs/injection-results.tsv"
|
|
[ -f "$tsv" ] || fail "controller check: $tsv not found -- inject_resources() must run first"
|
|
|
|
local name value result accepted_names=() skipped_names=()
|
|
while IFS=$'\t' read -r name value result; do
|
|
if [ "$result" = "accepted" ]; then
|
|
accepted_names+=("$name")
|
|
else
|
|
skipped_names+=("$name")
|
|
fi
|
|
done < "$tsv"
|
|
|
|
if [ "${#skipped_names[@]}" -gt 0 ]; then
|
|
log "controller check: skipping ${#skipped_names[@]} non-accepted resource(s) -- never actually applied at the controller, nothing to verify there: ${skipped_names[*]}"
|
|
fi
|
|
|
|
if [ "${#accepted_names[@]}" -eq 0 ]; then
|
|
log "controller check: no accepted resources to verify"
|
|
return 0
|
|
fi
|
|
|
|
# The live `resval` query already ran EARLIER, nested inside
|
|
# inject_resources()'s own xce_test worker session via a `!` shell
|
|
# escape (bug-1579 fix -- see internal_controller_check() above), while
|
|
# the synthetic worker's connection to the controller was still open.
|
|
# This function just reads back what that nested query already
|
|
# collected -- see the "Step 5" comment block above.
|
|
local err_txt="$TOP/logs/controller-check-error.txt"
|
|
if [ -f "$err_txt" ]; then
|
|
fail "controller check failed -- xce_test client context could not connect (masters=$PEERS); xce_test output:
|
|
$(cat "$err_txt")"
|
|
fi
|
|
|
|
local raw_tsv="$TOP/logs/controller-check-raw.tsv"
|
|
[ -f "$raw_tsv" ] || fail "controller check: $raw_tsv not found -- inject_resources()'s nested controller-check (internal_controller_check(), invoked via its xce_test session's \`!\` shell escape) did not run or did not complete"
|
|
|
|
log "controller check: reporting on ${#accepted_names[@]} accepted resource(s) (queried during injection, see internal_controller_check()): ${accepted_names[*]}"
|
|
|
|
local -A parsed_values=()
|
|
while IFS=$'\t' read -r name value; do
|
|
[ -n "$name" ] || continue
|
|
parsed_values["$name"]="$value"
|
|
done < "$raw_tsv"
|
|
|
|
mkdir -p "$TOP/logs"
|
|
local results_tsv="$TOP/logs/controller-check-results.tsv"
|
|
: > "$results_tsv"
|
|
|
|
local injected raw observed result_str
|
|
for name in "${accepted_names[@]}"; do
|
|
injected="${INJECTED_VALUES[$name]}"
|
|
raw="${parsed_values[$name]:-}"
|
|
|
|
if [ -z "$raw" ]; then
|
|
observed=""
|
|
result_str="MISMATCH: controller returned no value (injected=$injected observed=<none>)"
|
|
ANY_CONTROLLER_MISMATCH=1
|
|
else
|
|
# First whitespace-delimited token of the rendered value --
|
|
# strips a bytes-valued resource's trailing "(X.XX UNIT)"
|
|
# suffix (e.g. "8.500000 (8.50 B)" -> "8.500000"); a no-op for
|
|
# every other rendering (plain %f, %s, or true/false).
|
|
observed="$(awk '{print $1}' <<< "$raw")"
|
|
if controller_values_match "$injected" "$observed"; then
|
|
result_str="MATCH"
|
|
else
|
|
result_str="MISMATCH: injected=$injected observed=$observed"
|
|
ANY_CONTROLLER_MISMATCH=1
|
|
fi
|
|
fi
|
|
|
|
CONTROLLER_OBSERVED["$name"]="$observed"
|
|
CONTROLLER_CHECK_RESULTS["$name"]="$result_str"
|
|
log "controller check: $name -> $result_str"
|
|
printf '%s\t%s\t%s\t%s\n' "$name" "$injected" "$observed" "$result_str" >> "$results_tsv"
|
|
done
|
|
}
|
|
|
|
# ── Step 6: scheduler-side verification check (task 6.3) ───────────────────
|
|
# Requirement 4.1/4.2/4.3: for every SUCCESSFULLY-injected (accepted)
|
|
# resource/worker pair, poll (short interval, bounded generous total wait --
|
|
# design.md's "poll with a short interval up to a generous timeout, not
|
|
# tight polling that could flake") the live scheduler daemon's own log file
|
|
# for task 4.1's trace line -- "resval applied host=<H> resource=<R>
|
|
# value=<V>" at DEBUG3 (`-vvv`, already how start-all.sh starts the
|
|
# scheduler, per task 3.1) -- and compare the parsed value against the
|
|
# injected one, reusing controller_values_match() (numeric-with-tolerance vs.
|
|
# exact-string, same rule for either layer). A resource whose trace line
|
|
# never appears within the bounded wait is reported as "unreachable" /
|
|
# timeout, explicitly distinct from a value MISMATCH (design.md Error
|
|
# Handling -- "Layer unreachable" is its own category, not folded into
|
|
# mismatch). No record/replay/REPL round-trip -- a plain tail/grep of a live
|
|
# log file, per the System Flows sequence diagram. A REJECTED resource was
|
|
# never actually applied anywhere (task 2.2's own follow-up query inside
|
|
# resupdate already confirmed that at injection time) -- nothing to verify
|
|
# for it here either, same skip rule query_controller_resvals() (task 6.2)
|
|
# already established.
|
|
|
|
# Poll interval / bounded total wait, shared across the WHOLE batch of
|
|
# still-pending resources per iteration (see poll_scheduler_log below), not
|
|
# a separate full bounded wait PER resource -- so N accepted resources costs
|
|
# at most one ~15s bounded wait in the worst case (all N stuck), not N of
|
|
# them stacked sequentially; in the common case (scheduler already applied
|
|
# everything by the time this runs, since it fires after inject_resources()
|
|
# AND query_controller_resvals()'s own extra RPC round trip) it costs one
|
|
# fast pass with no sleep at all.
|
|
SCHED_POLL_INTERVAL="0.5"
|
|
SCHED_POLL_MAX_ITERS=30 # 30 x 0.5s = 15s total bounded wait
|
|
|
|
# poll_scheduler_log SCHED_LOG_FILE HOST NAME...
|
|
# Core bounded-wait poller, deliberately factored out of
|
|
# query_scheduler_resvals() so it can be exercised directly (e.g. against a
|
|
# name/host combination that was never actually injected, to prove the
|
|
# "unreachable" path is genuinely reachable) independent of the
|
|
# accepted-injection bookkeeping -- same "pure/self-contained, independently
|
|
# testable" spirit as controller_values_match(). Each iteration takes ONE
|
|
# full pass over SCHED_LOG_FILE (a single awk invocation building a
|
|
# name->value map for every "resval applied host=<H> resource=..." line
|
|
# seen for HOST, last occurrence per name wins) and checks every STILL-
|
|
# PENDING name against that map, rather than a separate grep-with-its-own-
|
|
# timeout per name run in sequence -- by the time a later name in the batch
|
|
# is checked, an earlier name's line has almost certainly already appeared
|
|
# in the same log-file read. Prints "name<TAB>value" to stdout for every
|
|
# name that WAS observed within the bounded wait; a name with no
|
|
# corresponding output line was NOT observed -- the caller's job to report
|
|
# that distinctly as "unreachable", not silently as a mismatch with an empty
|
|
# value.
|
|
poll_scheduler_log() {
|
|
local sched_log="$1" host="$2"
|
|
shift 2
|
|
local pending=("$@")
|
|
local -A found=()
|
|
local iter=0
|
|
|
|
while [ "$iter" -lt "$SCHED_POLL_MAX_ITERS" ] && [ "${#pending[@]}" -gt 0 ]; do
|
|
iter=$((iter + 1))
|
|
|
|
local name val
|
|
while IFS=$'\t' read -r name val; do
|
|
[ -n "$name" ] || continue
|
|
found["$name"]="$val"
|
|
done < <(awk -v needle="resval applied host=${host} resource=" '
|
|
index($0, needle) > 0 {
|
|
line = $0
|
|
sub(/^.*resource=/, "", line)
|
|
sep = index(line, " value=")
|
|
if (sep > 0) {
|
|
seen[substr(line, 1, sep - 1)] = substr(line, sep + 7)
|
|
}
|
|
}
|
|
END { for (n in seen) print n "\t" seen[n] }
|
|
' "$sched_log" 2>/dev/null)
|
|
|
|
local still_pending=()
|
|
for name in "${pending[@]}"; do
|
|
if [ -z "${found[$name]+x}" ]; then
|
|
still_pending+=("$name")
|
|
fi
|
|
done
|
|
pending=("${still_pending[@]}")
|
|
|
|
if [ "${#pending[@]}" -gt 0 ] && [ "$iter" -lt "$SCHED_POLL_MAX_ITERS" ]; then
|
|
sleep "$SCHED_POLL_INTERVAL"
|
|
fi
|
|
done
|
|
|
|
local n
|
|
for n in "${!found[@]}"; do
|
|
printf '%s\t%s\n' "$n" "${found[$n]}"
|
|
done
|
|
}
|
|
|
|
declare -A SCHEDULER_OBSERVED=() # resource name -> raw applied value string from the scheduler's trace log (only set for accepted names actually observed)
|
|
declare -A SCHEDULER_CHECK_RESULTS=() # resource name -> "MATCH" | "MISMATCH: injected=<X> observed=<Y>" | "UNREACHABLE: ..."
|
|
ANY_SCHEDULER_MISMATCH=0 # groundwork for task 7.1's verdict aggregation, mirrors ANY_CONTROLLER_MISMATCH; this task's own exit code does NOT depend on it
|
|
ANY_SCHEDULER_UNREACHABLE=0 # groundwork for task 7.1; kept distinct from ANY_SCHEDULER_MISMATCH per Requirement 4.2/4.3 and design.md's "Layer unreachable" being its own error category
|
|
|
|
# query_scheduler_resvals -- reads $TOP/logs/injection-results.tsv (same
|
|
# durable, order-preserving source query_controller_resvals() (task 6.2)
|
|
# reads) and, for every name marked "accepted", polls $TOP/logs/
|
|
# scheduler.log via poll_scheduler_log() above. Populates
|
|
# SCHEDULER_OBSERVED/SCHEDULER_CHECK_RESULTS and $TOP/logs/
|
|
# scheduler-check-results.tsv (name\tinjected\tobserved\tresult), following
|
|
# the exact same pattern query_controller_resvals() established for its own
|
|
# layer.
|
|
query_scheduler_resvals() {
|
|
local tsv="$TOP/logs/injection-results.tsv"
|
|
[ -f "$tsv" ] || fail "scheduler check: $tsv not found -- inject_resources() must run first"
|
|
|
|
local name value result accepted_names=() skipped_names=()
|
|
while IFS=$'\t' read -r name value result; do
|
|
if [ "$result" = "accepted" ]; then
|
|
accepted_names+=("$name")
|
|
else
|
|
skipped_names+=("$name")
|
|
fi
|
|
done < "$tsv"
|
|
|
|
if [ "${#skipped_names[@]}" -gt 0 ]; then
|
|
log "scheduler check: skipping ${#skipped_names[@]} non-accepted resource(s) -- never actually applied at the scheduler, nothing to verify there: ${skipped_names[*]}"
|
|
fi
|
|
|
|
if [ "${#accepted_names[@]}" -eq 0 ]; then
|
|
log "scheduler check: no accepted resources to verify"
|
|
return 0
|
|
fi
|
|
|
|
local sched_log="$TOP/logs/scheduler.log"
|
|
local total_wait
|
|
total_wait="$(awk -v i="$SCHED_POLL_INTERVAL" -v n="$SCHED_POLL_MAX_ITERS" 'BEGIN{printf "%.1f", i*n}')"
|
|
log "scheduler check: polling $sched_log for ${#accepted_names[@]} accepted resource(s), bounded wait up to ${total_wait}s total: ${accepted_names[*]}"
|
|
|
|
local -A observed_map=()
|
|
local n v
|
|
while IFS=$'\t' read -r n v; do
|
|
[ -n "$n" ] || continue
|
|
observed_map["$n"]="$v"
|
|
done < <(poll_scheduler_log "$sched_log" "$WORKER_HOST" "${accepted_names[@]}")
|
|
|
|
mkdir -p "$TOP/logs"
|
|
local results_tsv="$TOP/logs/scheduler-check-results.tsv"
|
|
: > "$results_tsv"
|
|
|
|
local injected observed result_str
|
|
for name in "${accepted_names[@]}"; do
|
|
injected="${INJECTED_VALUES[$name]}"
|
|
|
|
if [ -z "${observed_map[$name]+x}" ]; then
|
|
observed=""
|
|
result_str="UNREACHABLE: no scheduler trace-log line observed within bounded wait (injected=$injected observed=<none>)"
|
|
ANY_SCHEDULER_UNREACHABLE=1
|
|
else
|
|
observed="${observed_map[$name]}"
|
|
if controller_values_match "$injected" "$observed"; then
|
|
result_str="MATCH"
|
|
else
|
|
result_str="MISMATCH: injected=$injected observed=$observed"
|
|
ANY_SCHEDULER_MISMATCH=1
|
|
fi
|
|
fi
|
|
|
|
SCHEDULER_OBSERVED["$name"]="$observed"
|
|
SCHEDULER_CHECK_RESULTS["$name"]="$result_str"
|
|
log "scheduler check: $name -> $result_str"
|
|
printf '%s\t%s\t%s\t%s\n' "$name" "$injected" "$observed" "$result_str" >> "$results_tsv"
|
|
done
|
|
}
|
|
|
|
# ── Step 7: apiserver-side verification check (task 6.4) ───────────────────
|
|
# Requirement 5.1/5.2/5.4: for every SUCCESSFULLY-injected (accepted)
|
|
# resource/worker pair, poll (short interval, bounded generous total wait --
|
|
# same batched-polling design as query_scheduler_resvals()/task 6.3 above)
|
|
# the live apiserver's own log file for task 4.2's trace line -- a zap
|
|
# message "resource-location stored" with fields resource/host/value, e.g.
|
|
# `resource-location stored {"resource": "r15m", "host": "resverify-worker",
|
|
# "value": 1.5}` -- and compare the parsed value against the injected one,
|
|
# reusing controller_values_match() (numeric-with-tolerance vs. exact-string,
|
|
# same rule for every layer). A resource whose trace line never appears
|
|
# within the bounded wait is reported as "unreachable" / timeout, explicitly
|
|
# distinct from a value MISMATCH (design.md Error Handling -- "Layer
|
|
# unreachable" is its own category, not folded into mismatch). No xcecli
|
|
# query, no REST call -- a plain tail/grep of a live log file, per the System
|
|
# Flows sequence diagram. A REJECTED resource was never actually applied
|
|
# anywhere -- nothing to verify for it here either, same skip rule
|
|
# query_controller_resvals()/query_scheduler_resvals() already established.
|
|
|
|
# Poll interval / bounded total wait -- identical shape to
|
|
# SCHED_POLL_INTERVAL/SCHED_POLL_MAX_ITERS above (task 6.3), shared across
|
|
# the whole batch of still-pending resources per iteration (see
|
|
# poll_apiserver_log below), not a separate bounded wait per resource.
|
|
APISERVER_POLL_INTERVAL="0.5"
|
|
APISERVER_POLL_MAX_ITERS=30 # 30 x 0.5s = 15s total bounded wait
|
|
|
|
# poll_apiserver_log APISERVER_LOG_FILE HOST NAME...
|
|
# Core bounded-wait poller for the apiserver layer, mirroring
|
|
# poll_scheduler_log() above (task 6.3) in shape: factored out of
|
|
# query_apiserver_resvals() so it can be exercised directly (e.g. against a
|
|
# name/host combination that was never actually injected, to prove the
|
|
# "unreachable" path is genuinely reachable) independent of the
|
|
# accepted-injection bookkeeping. Each iteration takes ONE full pass over
|
|
# APISERVER_LOG_FILE (a single awk invocation building a name->value map for
|
|
# every "resource-location stored" line seen for HOST, last occurrence per
|
|
# name wins) and checks every STILL-PENDING name against that map, rather
|
|
# than a separate grep-with-its-own-timeout per name. Prints "name<TAB>value"
|
|
# to stdout for every name that WAS observed within the bounded wait; a name
|
|
# with no corresponding output line was NOT observed -- the caller's job to
|
|
# report that distinctly as "unreachable", not silently as a mismatch with an
|
|
# empty value.
|
|
#
|
|
# Line format (task 4.2, handler.go's OnResourceLocationUpdated, zap console
|
|
# encoding -- confirmed live against this script's own apiserver.log output):
|
|
# the message is followed by a trailing JSON object holding the structured
|
|
# fields, in a FIXED field order (resource, host, value) -- "value" renders
|
|
# unquoted for numeric/boolean resources (float64/bool zap fields, this
|
|
# script's default set) and double-quoted for string-typed resources
|
|
# (type/model, excluded from the default set per bug-1574). Parsed below via
|
|
# index()/substr()/sub() -- not a JSON library -- mirroring
|
|
# poll_scheduler_log()'s own non-gawk-specific approach for its fixed-format
|
|
# trace line; the trailing quote pair (if present) is stripped so the
|
|
# numeric-vs-string branch in controller_values_match() sees the same shape
|
|
# either way.
|
|
poll_apiserver_log() {
|
|
local api_log="$1" host="$2"
|
|
shift 2
|
|
local pending=("$@")
|
|
local -A found=()
|
|
local iter=0
|
|
|
|
while [ "$iter" -lt "$APISERVER_POLL_MAX_ITERS" ] && [ "${#pending[@]}" -gt 0 ]; do
|
|
iter=$((iter + 1))
|
|
|
|
local name val
|
|
while IFS=$'\t' read -r name val; do
|
|
[ -n "$name" ] || continue
|
|
found["$name"]="$val"
|
|
done < <(awk -v host="$host" '
|
|
index($0, "resource-location stored") > 0 && index($0, "\"host\": \"" host "\"") > 0 {
|
|
line = $0
|
|
|
|
rtag = "\"resource\": \""
|
|
rstart = index(line, rtag)
|
|
if (rstart == 0) next
|
|
rest = substr(line, rstart + length(rtag))
|
|
rend = index(rest, "\"")
|
|
if (rend == 0) next
|
|
name = substr(rest, 1, rend - 1)
|
|
|
|
vtag = "\"value\": "
|
|
vstart = index(line, vtag)
|
|
if (vstart == 0) next
|
|
vraw = substr(line, vstart + length(vtag))
|
|
sub(/}[ \r]*$/, "", vraw)
|
|
if (length(vraw) >= 2 && substr(vraw, 1, 1) == "\"" && substr(vraw, length(vraw), 1) == "\"") {
|
|
vraw = substr(vraw, 2, length(vraw) - 2)
|
|
}
|
|
seen[name] = vraw
|
|
}
|
|
END { for (n in seen) print n "\t" seen[n] }
|
|
' "$api_log" 2>/dev/null)
|
|
|
|
local still_pending=()
|
|
for name in "${pending[@]}"; do
|
|
if [ -z "${found[$name]+x}" ]; then
|
|
still_pending+=("$name")
|
|
fi
|
|
done
|
|
pending=("${still_pending[@]}")
|
|
|
|
if [ "${#pending[@]}" -gt 0 ] && [ "$iter" -lt "$APISERVER_POLL_MAX_ITERS" ]; then
|
|
sleep "$APISERVER_POLL_INTERVAL"
|
|
fi
|
|
done
|
|
|
|
local n
|
|
for n in "${!found[@]}"; do
|
|
printf '%s\t%s\n' "$n" "${found[$n]}"
|
|
done
|
|
}
|
|
|
|
declare -A APISERVER_OBSERVED=() # resource name -> raw stored value string from the apiserver's trace log (only set for accepted names actually observed)
|
|
declare -A APISERVER_CHECK_RESULTS=() # resource name -> "MATCH" | "MISMATCH: injected=<X> observed=<Y>" | "UNREACHABLE: ..."
|
|
ANY_APISERVER_MISMATCH=0 # groundwork for task 7.1's verdict aggregation, mirrors ANY_CONTROLLER_MISMATCH/ANY_SCHEDULER_MISMATCH; this task's own exit code does NOT depend on it
|
|
ANY_APISERVER_UNREACHABLE=0 # groundwork for task 7.1, mirrors ANY_SCHEDULER_UNREACHABLE; kept distinct from ANY_APISERVER_MISMATCH per Requirement 5.2/5.4 and design.md's "Layer unreachable" being its own error category
|
|
|
|
# query_apiserver_resvals -- reads $TOP/logs/injection-results.tsv (same
|
|
# durable, order-preserving source query_controller_resvals()/
|
|
# query_scheduler_resvals() read) and, for every name marked "accepted",
|
|
# polls $TOP/logs/apiserver.log via poll_apiserver_log() above. Populates
|
|
# APISERVER_OBSERVED/APISERVER_CHECK_RESULTS and $TOP/logs/
|
|
# apiserver-check-results.tsv (name\tinjected\tobserved\tresult), following
|
|
# the exact same pattern query_scheduler_resvals() (task 6.3) established for
|
|
# its own layer.
|
|
query_apiserver_resvals() {
|
|
local tsv="$TOP/logs/injection-results.tsv"
|
|
[ -f "$tsv" ] || fail "apiserver check: $tsv not found -- inject_resources() must run first"
|
|
|
|
local name value result accepted_names=() skipped_names=()
|
|
while IFS=$'\t' read -r name value result; do
|
|
if [ "$result" = "accepted" ]; then
|
|
accepted_names+=("$name")
|
|
else
|
|
skipped_names+=("$name")
|
|
fi
|
|
done < "$tsv"
|
|
|
|
if [ "${#skipped_names[@]}" -gt 0 ]; then
|
|
log "apiserver check: skipping ${#skipped_names[@]} non-accepted resource(s) -- never actually applied at the apiserver, nothing to verify there: ${skipped_names[*]}"
|
|
fi
|
|
|
|
if [ "${#accepted_names[@]}" -eq 0 ]; then
|
|
log "apiserver check: no accepted resources to verify"
|
|
return 0
|
|
fi
|
|
|
|
local api_log="$TOP/logs/apiserver.log"
|
|
local total_wait
|
|
total_wait="$(awk -v i="$APISERVER_POLL_INTERVAL" -v n="$APISERVER_POLL_MAX_ITERS" 'BEGIN{printf "%.1f", i*n}')"
|
|
log "apiserver check: polling $api_log for ${#accepted_names[@]} accepted resource(s), bounded wait up to ${total_wait}s total: ${accepted_names[*]}"
|
|
|
|
local -A observed_map=()
|
|
local n v
|
|
while IFS=$'\t' read -r n v; do
|
|
[ -n "$n" ] || continue
|
|
observed_map["$n"]="$v"
|
|
done < <(poll_apiserver_log "$api_log" "$WORKER_HOST" "${accepted_names[@]}")
|
|
|
|
mkdir -p "$TOP/logs"
|
|
local results_tsv="$TOP/logs/apiserver-check-results.tsv"
|
|
: > "$results_tsv"
|
|
|
|
local injected observed result_str
|
|
for name in "${accepted_names[@]}"; do
|
|
injected="${INJECTED_VALUES[$name]}"
|
|
|
|
if [ -z "${observed_map[$name]+x}" ]; then
|
|
observed=""
|
|
result_str="UNREACHABLE: no apiserver trace-log line observed within bounded wait (injected=$injected observed=<none>)"
|
|
ANY_APISERVER_UNREACHABLE=1
|
|
else
|
|
observed="${observed_map[$name]}"
|
|
if controller_values_match "$injected" "$observed"; then
|
|
result_str="MATCH"
|
|
else
|
|
result_str="MISMATCH: injected=$injected observed=$observed"
|
|
ANY_APISERVER_MISMATCH=1
|
|
fi
|
|
fi
|
|
|
|
APISERVER_OBSERVED["$name"]="$observed"
|
|
APISERVER_CHECK_RESULTS["$name"]="$result_str"
|
|
log "apiserver check: $name -> $result_str"
|
|
printf '%s\t%s\t%s\t%s\n' "$name" "$injected" "$observed" "$result_str" >> "$results_tsv"
|
|
done
|
|
}
|
|
|
|
# ── Step 8: verdict aggregation, VERDICT line, PASS/FAIL oracle (task 7.1) ──
|
|
# Requirement 6.2/6.3/6.4: combine the controller/scheduler/apiserver
|
|
# per-layer results (tasks 6.2-6.4) plus the injection accept/reject outcome
|
|
# (tasks 2.1/2.2/6.1) into a single overall verdict. Deliberately
|
|
# implemented as a SELF-CONTAINED comparison/aggregation function --
|
|
# aggregate_verdict() below takes an array of plain TSV "records" (one per
|
|
# checked resource, see the format comment on aggregate_verdict() itself)
|
|
# and does its own MATCH/MISMATCH/UNREACHABLE classification via
|
|
# controller_values_match() (task 6.2's own pure comparison helper, reused
|
|
# here rather than re-parsing/re-trusting each layer's already-rendered
|
|
# CHECK_RESULTS string) -- it runs no live command, reads no file, and
|
|
# touches no global deployment state, so it is directly callable with
|
|
# hand-written SYNTHETIC records for testing (this task's own sanity-check
|
|
# invocations during implementation, and task 8.2's dedicated harness, which
|
|
# depends on this exact function/record format). main() below builds the
|
|
# record array from the ALREADY-POPULATED INJECTED_VALUES/INJECTED_RESULTS
|
|
# (task 6.1) and CONTROLLER_OBSERVED/SCHEDULER_OBSERVED/APISERVER_OBSERVED
|
|
# (tasks 6.2-6.4) associative arrays for the live-run path; a synthetic
|
|
# caller builds the same shape by hand, with no deployment involved at all.
|
|
#
|
|
# Per design.md's Error Handling, a REJECTED resource name (Requirement 1.2)
|
|
# is a run-level failure in its own right -- reported as its own failure
|
|
# category ("REJECTED", distinct from a per-layer MISMATCH/UNREACHABLE) and
|
|
# always drives the overall verdict to FAIL, independent of the (nonexistent
|
|
# -- a rejected name is never actually applied anywhere, see
|
|
# query_controller_resvals()/query_scheduler_resvals()/
|
|
# query_apiserver_resvals()'s shared skip rule) per-layer results.
|
|
|
|
# Record format (9 fields, separated by VERDICT_FIELD_SEP -- ASCII Unit
|
|
# Separator 0x1F, deliberately NOT a tab, see that constant's own comment
|
|
# below), one record string per checked resource:
|
|
# resource<SEP>injected<SEP>injection_status<SEP>
|
|
# ctrl_reachable<SEP>ctrl_observed<SEP>
|
|
# sched_reachable<SEP>sched_observed<SEP>
|
|
# api_reachable<SEP>api_observed
|
|
#
|
|
# resource -- resource name.
|
|
# injected -- the value that was (or would have been) injected.
|
|
# injection_status -- "accepted" or "rejected: <reason>"
|
|
# (INJECTED_RESULTS' own vocabulary, tasks 2.2/6.1)
|
|
# -- anything other than the literal string
|
|
# "accepted" is treated as a rejection, independent
|
|
# of the reason text.
|
|
# *_reachable -- "1" if that layer produced an observed value
|
|
# within scope/bounded wait, "0" if it did not
|
|
# (timeout, or simply never checked). Ignored when
|
|
# injection_status != "accepted" -- nothing was ever
|
|
# applied anywhere for a rejected name, so no layer
|
|
# is "checked" for it at all (matches tasks
|
|
# 6.2-6.4's own skip rule).
|
|
# *_observed -- observed value string for that layer, meaningful
|
|
# only when the paired *_reachable flag is "1".
|
|
#
|
|
# A record for a rejected resource still needs all 9 fields present
|
|
# (positional read) -- pass "0"/"" for each of the 6 layer fields by
|
|
# convention (see build_verdict_record() below); their content is ignored.
|
|
#
|
|
# VERDICT_FIELD_SEP: ASCII Unit Separator (0x1F), NOT a tab. Found live
|
|
# (this task's own synthetic sanity-check run, scenario (c) below) that a
|
|
# plain `IFS=$'\t' read` -- the convention every OTHER TSV in this file
|
|
# uses -- silently COLLAPSES a genuinely-empty field: bash treats tab (like
|
|
# space/newline) as "IFS whitespace" for splitting purposes even when IFS is
|
|
# set to ONLY a tab character, so two adjacent delimiters (an empty
|
|
# *_observed field, which legitimately happens both for an UNREACHABLE
|
|
# layer and for a reachable-but-empty-value MISMATCH, e.g. controller "no
|
|
# value" -- see query_controller_resvals()) collapse into one, shifting
|
|
# every field after it and silently corrupting the record. None of this
|
|
# file's other TSVs (injection-results.tsv, *-check-results.tsv) ever carry
|
|
# a genuinely-empty field, which is why this gotcha never surfaced before
|
|
# aggregate_verdict()'s record format, the first place in this script where
|
|
# an empty field is a legitimate, expected value. 0x1F is not "IFS
|
|
# whitespace" and is never expected to appear in a resource name, value, or
|
|
# status string, so it splits/preserves empty fields correctly with a plain
|
|
# `IFS=$VERDICT_FIELD_SEP read`.
|
|
VERDICT_FIELD_SEP=$'\x1f'
|
|
|
|
build_verdict_record() {
|
|
# build_verdict_record NAME INJECTED INJ_STATUS CTRL_OK CTRL_OBS SCHED_OK SCHED_OBS API_OK API_OBS
|
|
# Thin formatting helper so main()'s live-run wiring and any hand-built
|
|
# synthetic record share one place that defines field order/count -- not
|
|
# itself part of the "self-contained comparison step" (aggregate_verdict()
|
|
# is that step); a caller may also just printf/construct the
|
|
# VERDICT_FIELD_SEP-joined string directly, this exists purely for
|
|
# readability at call sites.
|
|
local sep="$VERDICT_FIELD_SEP"
|
|
printf '%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s' \
|
|
"$1" "$sep" "$2" "$sep" "$3" "$sep" "$4" "$sep" "$5" "$sep" \
|
|
"$6" "$sep" "$7" "$sep" "$8" "$sep" "$9"
|
|
}
|
|
|
|
# aggregate_verdict RECORD...
|
|
# THE self-contained comparison/aggregation step (Requirement 6.2/6.3/6.4).
|
|
# Pure function over its arguments: no network I/O, no file reads, no
|
|
# subprocess -- every classification decision is either a direct field
|
|
# comparison or a call to controller_values_match() (also pure). Safe and
|
|
# deterministic to call with hand-built synthetic records, live-run-derived
|
|
# records, or a mix, with no live deployment required either way.
|
|
#
|
|
# Resets and repopulates these globals on every call (so repeated calls in
|
|
# the same shell/test session -- e.g. one per synthetic scenario -- never
|
|
# see a stale result from a previous invocation):
|
|
# AGG_RESOURCE_STATUS["<name>"] = "PASS" | "FAIL"
|
|
# AGG_FAILURE_LINES[] = one human-readable "resource=... layer=...
|
|
# status=... ..." string per failing
|
|
# (resource, layer) pair or rejected name
|
|
# AGG_TOTAL_RESOURCES = count of records passed in
|
|
# AGG_MISMATCHES = count of (resource, layer) pairs
|
|
# classified MISMATCH
|
|
# AGG_UNREACHABLE = count of (resource, layer) pairs
|
|
# classified UNREACHABLE
|
|
# AGG_REJECTED = count of records whose injection_status
|
|
# was not "accepted"
|
|
# AGG_RESULT = "PASS" | "FAIL"
|
|
#
|
|
# Prints one FAILURE line per failing (resource, layer) pair or rejected
|
|
# name (naming the specific resource + layer, per Requirement 6.3), then
|
|
# exactly one VERDICT line, then exactly one terminal PASS/FAIL oracle line
|
|
# -- xce-test-ha's machine-parseable convention, see SKILL.md.
|
|
#
|
|
# Returns (exit code) 0 when AGG_RESULT=PASS, 1 when AGG_RESULT=FAIL -- so a
|
|
# caller (main(), or a synthetic test) can branch directly on the call
|
|
# itself: `if aggregate_verdict "${records[@]}"; then ... fi`.
|
|
declare -A AGG_RESOURCE_STATUS=()
|
|
AGG_FAILURE_LINES=()
|
|
AGG_TOTAL_RESOURCES=0
|
|
AGG_MISMATCHES=0
|
|
AGG_UNREACHABLE=0
|
|
AGG_REJECTED=0
|
|
AGG_RESULT="PASS"
|
|
|
|
aggregate_verdict() {
|
|
AGG_RESOURCE_STATUS=()
|
|
AGG_FAILURE_LINES=()
|
|
AGG_TOTAL_RESOURCES=0
|
|
AGG_MISMATCHES=0
|
|
AGG_UNREACHABLE=0
|
|
AGG_REJECTED=0
|
|
AGG_RESULT="PASS"
|
|
|
|
local rec resource injected inj_status
|
|
local ctrl_ok ctrl_obs sched_ok sched_obs api_ok api_obs
|
|
local layer ok obs resource_failed
|
|
|
|
for rec in "$@"; do
|
|
IFS="$VERDICT_FIELD_SEP" read -r resource injected inj_status \
|
|
ctrl_ok ctrl_obs sched_ok sched_obs api_ok api_obs <<< "$rec"
|
|
|
|
AGG_TOTAL_RESOURCES=$((AGG_TOTAL_RESOURCES + 1))
|
|
resource_failed=0
|
|
|
|
if [ "$inj_status" != "accepted" ]; then
|
|
AGG_REJECTED=$((AGG_REJECTED + 1))
|
|
resource_failed=1
|
|
AGG_FAILURE_LINES+=("resource=$resource layer=injection status=REJECTED detail=\"$inj_status\"")
|
|
else
|
|
for layer in controller scheduler apiserver; do
|
|
case "$layer" in
|
|
controller) ok="$ctrl_ok"; obs="$ctrl_obs" ;;
|
|
scheduler) ok="$sched_ok"; obs="$sched_obs" ;;
|
|
apiserver) ok="$api_ok"; obs="$api_obs" ;;
|
|
esac
|
|
|
|
if [ "$ok" != "1" ]; then
|
|
AGG_UNREACHABLE=$((AGG_UNREACHABLE + 1))
|
|
resource_failed=1
|
|
AGG_FAILURE_LINES+=("resource=$resource layer=$layer status=UNREACHABLE injected=$injected observed=<none>")
|
|
elif ! controller_values_match "$injected" "$obs"; then
|
|
AGG_MISMATCHES=$((AGG_MISMATCHES + 1))
|
|
resource_failed=1
|
|
AGG_FAILURE_LINES+=("resource=$resource layer=$layer status=MISMATCH injected=$injected observed=$obs")
|
|
fi
|
|
done
|
|
fi
|
|
|
|
if [ "$resource_failed" -eq 1 ]; then
|
|
AGG_RESOURCE_STATUS["$resource"]="FAIL"
|
|
AGG_RESULT="FAIL"
|
|
else
|
|
AGG_RESOURCE_STATUS["$resource"]="PASS"
|
|
fi
|
|
done
|
|
|
|
if [ "$AGG_TOTAL_RESOURCES" -eq 0 ]; then
|
|
# No resources at all is not a legitimate PASS -- nothing was
|
|
# actually verified. Structural/degenerate case, not expected from
|
|
# main()'s own live-run wiring (inject_resources() already fail()s
|
|
# the run before this point if RESOURCE_ARGS somehow resolved empty
|
|
# -- DEFAULT_RESOURCE_ARGS is always non-empty), but a synthetic
|
|
# caller could legitimately probe this edge, so classify it
|
|
# explicitly rather than silently reporting PASS on zero input.
|
|
AGG_RESULT="FAIL"
|
|
AGG_FAILURE_LINES+=("no resources were checked -- nothing to verify")
|
|
fi
|
|
|
|
local line
|
|
for line in "${AGG_FAILURE_LINES[@]}"; do
|
|
log "FAILURE: $line"
|
|
done
|
|
|
|
log "VERDICT resources=$AGG_TOTAL_RESOURCES layers_checked=3 mismatches=$AGG_MISMATCHES unreachable=$AGG_UNREACHABLE rejected=$AGG_REJECTED result=$AGG_RESULT"
|
|
|
|
if [ "$AGG_RESULT" = "PASS" ]; then
|
|
log "PASS: $AGG_TOTAL_RESOURCES resource(s), 3 layers checked, zero mismatches, zero unreachable, zero rejected"
|
|
return 0
|
|
else
|
|
log "FAIL: $AGG_TOTAL_RESOURCES resource(s), 3 layers checked, $AGG_MISMATCHES mismatch(es), $AGG_UNREACHABLE unreachable, $AGG_REJECTED rejected -- see FAILURE lines above"
|
|
return 1
|
|
fi
|
|
}
|
|
|
|
# build_verdict_records_from_tsv TSV_PATH
|
|
# Round-2 remediation of the round-1 REJECTED verdict (see the sched_ok/
|
|
# api_ok derivation comment inside for the actual bug/fix). This is main()'s
|
|
# own record-building loop (Task 7.1), extracted verbatim from main() into a
|
|
# standalone function so it is independently callable -- the round-1
|
|
# reviewer's exact finding was that a hand-built-synthetic-record test of
|
|
# aggregate_verdict() alone (task 8.2's harness) could never have caught this
|
|
# bug, because the bug lived entirely in main()'s data-marshaling code
|
|
# (reading $TOP/logs/injection-results.tsv plus the CONTROLLER_OBSERVED/
|
|
# SCHEDULER_OBSERVED/SCHEDULER_CHECK_RESULTS/APISERVER_OBSERVED/
|
|
# APISERVER_CHECK_RESULTS globals into aggregate_verdict()-shaped records),
|
|
# which a bypass-and-call-aggregate_verdict()-directly test never exercises.
|
|
# Extracting it here means a synthetic test can source this file, hand-
|
|
# populate the same globals + a synthetic TSV, call this function directly,
|
|
# and inspect its output -- exercising the EXACT SAME code main() itself
|
|
# runs, not a reimplementation of it. No behavior change from the inline
|
|
# version: same reads, same globals, same build_verdict_record() calls.
|
|
#
|
|
# Populates the global VERDICT_RECORDS array (reset on every call, mirroring
|
|
# aggregate_verdict()'s own reset-globals-on-every-call convention) with one
|
|
# build_verdict_record()-shaped record per line of TSV_PATH. Build one record
|
|
# per INJECTED name, in injection order -- read back from
|
|
# $TOP/logs/injection-results.tsv, the same durable, order-preserving source
|
|
# query_controller_resvals()/query_scheduler_resvals()/
|
|
# query_apiserver_resvals() already read (bash associative arrays have no
|
|
# defined iteration order, the TSV does). ANY_CONTROLLER_MISMATCH/
|
|
# ANY_SCHEDULER_MISMATCH/ANY_SCHEDULER_UNREACHABLE/ANY_APISERVER_MISMATCH/
|
|
# ANY_APISERVER_UNREACHABLE (set by the three query_*_resvals() calls) are
|
|
# NOT consulted here -- aggregate_verdict() does its own independent
|
|
# MATCH/MISMATCH/UNREACHABLE classification straight from the *_OBSERVED
|
|
# values via controller_values_match(), the same pure comparison helper each
|
|
# query_*_resvals() already uses, so a synthetic caller and this live-run
|
|
# path share identical classification logic.
|
|
VERDICT_RECORDS=()
|
|
build_verdict_records_from_tsv() {
|
|
local verdict_tsv="$1"
|
|
[ -f "$verdict_tsv" ] || fail "verdict aggregation: $verdict_tsv not found -- inject_resources() must run first"
|
|
|
|
VERDICT_RECORDS=()
|
|
local vname vvalue vstatus
|
|
local ctrl_ok ctrl_obs sched_ok sched_obs api_ok api_obs
|
|
while IFS=$'\t' read -r vname vvalue vstatus; do
|
|
[ -n "$vname" ] || continue
|
|
if [ "$vstatus" = "accepted" ]; then
|
|
# ctrl_ok: derived from CONTROLLER_OBSERVED array-key existence.
|
|
# This is fine as-is (round-2 review confirmed no change needed)
|
|
# -- query_controller_resvals()/internal_controller_check() has
|
|
# no separate UNREACHABLE concept at all; a query failure there
|
|
# aborts the whole run via fail() rather than leaving a
|
|
# placeholder entry, so "no observed value" legitimately means
|
|
# MISMATCH for this layer specifically, and array-key existence
|
|
# is an accurate proxy for "was queried".
|
|
if [ -n "${CONTROLLER_OBSERVED[$vname]+x}" ]; then
|
|
ctrl_ok=1; ctrl_obs="${CONTROLLER_OBSERVED[$vname]}"
|
|
else
|
|
ctrl_ok=0; ctrl_obs=""
|
|
fi
|
|
# sched_ok/api_ok: round-2 fix (remediation of the round-1
|
|
# REJECTED verdict). query_scheduler_resvals()/
|
|
# query_apiserver_resvals() (tasks 6.3/6.4) unconditionally set
|
|
# SCHEDULER_OBSERVED[$vname]/APISERVER_OBSERVED[$vname] for
|
|
# EVERY accepted+checked name, including on a genuine
|
|
# UNREACHABLE/timeout -- where the value is just set to "" (see
|
|
# the `observed=""` branch in both functions). So deriving
|
|
# sched_ok/api_ok from *_OBSERVED array-key existence (the old,
|
|
# rejected approach) was ALWAYS 1 for a checked name -- it could
|
|
# never see the genuine UNREACHABLE status, and aggregate_verdict
|
|
# fell through to comparing the injected value against an empty
|
|
# string, misreporting UNREACHABLE as MISMATCH. Fix: derive
|
|
# sched_ok/api_ok from SCHEDULER_CHECK_RESULTS[$vname]/
|
|
# APISERVER_CHECK_RESULTS[$vname] instead -- both are always
|
|
# populated (same unconditional-set guarantee, task 6.3/6.4) with
|
|
# one of exactly "MATCH", "MISMATCH: injected=... observed=...",
|
|
# or "UNREACHABLE: ..." (see the declare -A comments above each
|
|
# array), so a leading "UNREACHABLE:" is the correct, exact
|
|
# signal -- not the presence/absence of the OBSERVED entry.
|
|
# The `__UNCHECKED__` fallback below is defensive-only (not
|
|
# expected to trigger given query_scheduler_resvals()/
|
|
# query_apiserver_resvals() iterate the exact same accepted-name
|
|
# list this loop reads from the same injection-results.tsv, so
|
|
# CHECK_RESULTS is always populated by the time main() reaches
|
|
# this loop) -- but if it ever DID trigger, treating "never
|
|
# checked" as ok=1/empty-observed would silently reproduce this
|
|
# exact bug class, so it's treated as unreachable instead.
|
|
case "${SCHEDULER_CHECK_RESULTS[$vname]-__UNCHECKED__}" in
|
|
UNREACHABLE:*|__UNCHECKED__) sched_ok=0; sched_obs="" ;;
|
|
*) sched_ok=1; sched_obs="${SCHEDULER_OBSERVED[$vname]-}" ;;
|
|
esac
|
|
case "${APISERVER_CHECK_RESULTS[$vname]-__UNCHECKED__}" in
|
|
UNREACHABLE:*|__UNCHECKED__) api_ok=0; api_obs="" ;;
|
|
*) api_ok=1; api_obs="${APISERVER_OBSERVED[$vname]-}" ;;
|
|
esac
|
|
else
|
|
ctrl_ok=0; ctrl_obs=""
|
|
sched_ok=0; sched_obs=""
|
|
api_ok=0; api_obs=""
|
|
fi
|
|
VERDICT_RECORDS+=("$(build_verdict_record "$vname" "$vvalue" "$vstatus" \
|
|
"$ctrl_ok" "$ctrl_obs" "$sched_ok" "$sched_obs" "$api_ok" "$api_obs")")
|
|
done < "$verdict_tsv"
|
|
}
|
|
|
|
main() {
|
|
parse_args "$@"
|
|
resolve_top
|
|
|
|
# Trap registered before the FIRST process is started (deploy() below
|
|
# only generates scripts/certs/config; start_all() is what actually
|
|
# starts backgrounded processes) -- this covers a mid-deployment abort
|
|
# (e.g. apiserver fails to start after controller+scheduler are already
|
|
# up), not just a completed PASS/FAIL run. See Requirement 6.5 and
|
|
# design.md's Error Handling / Deployment failure entry.
|
|
register_trap
|
|
|
|
precondition_checks
|
|
deploy
|
|
start_all
|
|
confirm_deployment_up
|
|
|
|
inject_resources "${RESOURCE_ARGS[@]}"
|
|
|
|
query_controller_resvals
|
|
|
|
query_scheduler_resvals
|
|
|
|
query_apiserver_resvals
|
|
|
|
# --- Task 7.1: aggregate the controller/scheduler/apiserver per-layer
|
|
# results plus the injection accept/reject outcome into one VERDICT
|
|
# line + terminal PASS/FAIL oracle line (aggregate_verdict() above).
|
|
# build_verdict_records_from_tsv() (round-2 extraction, see its own
|
|
# header comment) does the data-marshaling; main() is just a thin
|
|
# caller. ---
|
|
build_verdict_records_from_tsv "$TOP/logs/injection-results.tsv"
|
|
|
|
log "aggregating verdict across ${#VERDICT_RECORDS[@]} resource(s) under $TOP; cleanup (trap-based, same mechanism as a mid-deployment abort) fires on exit below regardless of PASS/FAIL, unless --keep-running was passed -- logs under $TOP/logs are preserved either way"
|
|
|
|
if aggregate_verdict "${VERDICT_RECORDS[@]}"; then
|
|
exit 0
|
|
else
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
# Hidden internal entry point (bug-1579 fix): inject_resources()'s `!`
|
|
# shell-escape line invokes this same script as `bash "$SELF_SCRIPT"
|
|
# --internal-controller-check TOP PEERS WORKER_HOST NAME...` from inside its
|
|
# own xce_test worker session. Intercepted here, BEFORE main()/parse_args()
|
|
# (which has no knowledge of this flag and would otherwise reject it as an
|
|
# "Unknown argument") -- runs only internal_controller_check(), none of the
|
|
# normal deploy/inject/trap flow.
|
|
#
|
|
# Guarded so this file can be `source`d (e.g. a synthetic test harness that
|
|
# wants aggregate_verdict()/controller_values_match()/build_verdict_record()
|
|
# in scope, task 8.2's use case) without executing main() or
|
|
# internal_controller_check() as a side effect -- ${BASH_SOURCE[0]} (this
|
|
# file's own path) only equals ${0} when the script is the thing actually
|
|
# being run (directly executed, or `bash run-resource-verify.sh ...`,
|
|
# including the `--internal-controller-check` recursive invocation above,
|
|
# which is a fresh `bash "$SELF_SCRIPT" ...` process, not a source), not
|
|
# when it's sourced into another shell/script.
|
|
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
|
if [ "${1:-}" = "--internal-controller-check" ]; then
|
|
shift
|
|
internal_controller_check "$@"
|
|
exit $?
|
|
fi
|
|
|
|
main "$@"
|
|
fi
|