feat(resource-value-verification): wire deployment into skill via start-all.sh (task 5.1)
run-resource-verify.sh deploys controller+scheduler(-vvv)+apiserver(debug logging) via xce-deploy-ha's deploy-ha.sh -> start-all.sh into a fresh scratch dir, confirms all three are genuinely up, and cleans up via a trap registered before any process starts - covering both normal completion and a mid-deployment abort. Logs always preserved regardless of --keep-running. Independently stress-tested with 8+ SIGTERM interrupts at varying deployment points and a genuine crashing-apiserver scenario - zero leftover processes in every case. Single-node by default (propagation testing, not HA failover), overridable via $XCE_RESVERIFY_PEERS. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
2ca25eebd0
commit
da41cce2ea
SSH key fingerprint:
SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
3 changed files with 361 additions and 31 deletions
|
|
@ -6,34 +6,41 @@ model: haiku
|
|||
|
||||
# XCE Resource-Value Verification
|
||||
|
||||
## Status: scaffold only, not yet functional
|
||||
## Status: deployment step functional; injection/verification not yet built
|
||||
|
||||
This `SKILL.md` is task 1.2 of the `resource-value-verification` spec
|
||||
(`.kiro/specs/resource-value-verification/`) — a documentation-only stub that
|
||||
makes this skill **discoverable and invocable** and pins its contract ahead of
|
||||
implementation. **`run-resource-verify.sh`, the orchestration script this file
|
||||
describes, does not exist yet.** Invoking `/xce-test-resources` today will not
|
||||
run a live verification — follow this file to confirm that, and report the gap
|
||||
rather than fabricate a result.
|
||||
`run-resource-verify.sh` now exists (task 5.1) and implements the
|
||||
**deployment step only**: it invokes `xce-deploy-ha`'s `deploy-ha.sh` into a
|
||||
fresh scratch directory, then its generated `start-all.sh`, confirms
|
||||
controller + scheduler (`-vvv`) + apiserver (debug logging) are all
|
||||
genuinely up, and cleans up (trap-based, covers a mid-deployment abort, not
|
||||
just a completed run — see Error handling below). Invoking
|
||||
`/xce-test-resources` today runs a real deploy-confirm-cleanup cycle but
|
||||
does **not yet** inject a resource value or check any layer — it exits 0 on
|
||||
a successful deployment/cleanup and non-zero on a deployment failure, not on
|
||||
an injection PASS/FAIL verdict. Report this gap rather than fabricate a
|
||||
verdict if asked to check actual resource-value propagation.
|
||||
|
||||
The real orchestration logic is built incrementally by later tasks in the same
|
||||
spec:
|
||||
The rest of the orchestration logic is built incrementally by later tasks in
|
||||
the same spec:
|
||||
- **Task 2.1/2.2** — `resupdate` accepts any registered resource name and
|
||||
reports per-name accept/reject (closes the gap this whole feature exists to
|
||||
close; `resval_typeresolve.c`'s fallback path).
|
||||
close; `resval_typeresolve.c`'s fallback path). Done.
|
||||
- **Task 3.1** — `xce-deploy-ha`'s `deploy-ha.sh` grows a conditional apiserver
|
||||
step + a combined `start-all.sh` (controller → scheduler → apiserver).
|
||||
step + a combined `start-all.sh` (controller → scheduler → apiserver). Done.
|
||||
- **Task 4.1/4.2** — one trace-log line each in the scheduler
|
||||
(`core_events.c`, `XCE_CORE_EVENT_RESVALS_DELTA`) and the apiserver
|
||||
(`handler.go`'s `OnResourceLocationUpdated`) — what this skill will
|
||||
tail/grep for.
|
||||
tail/grep for. Done.
|
||||
- **Task 5.1** — wires `start-all.sh` + trap-based cleanup into this skill.
|
||||
Done (this is the deployment step described above).
|
||||
- **Task 6.1–6.4** — worker join + injection, and the controller/scheduler/
|
||||
apiserver checks themselves.
|
||||
apiserver checks themselves. Not yet built.
|
||||
- **Task 7.1** — verdict aggregation, the VERDICT line, and cleanup-on-every-path.
|
||||
Not yet built.
|
||||
|
||||
Once those land, `run-resource-verify.sh` will exist alongside this file and
|
||||
this section (and the rest of this document's hedging) should be removed.
|
||||
Once tasks 6.1–7.1 land, this section (and the rest of this document's
|
||||
hedging about the VERDICT line/PASS-FAIL oracle being aspirational) should
|
||||
be removed.
|
||||
|
||||
## What this skill will do (target behavior)
|
||||
|
||||
|
|
@ -78,21 +85,38 @@ mutable state carried across runs.
|
|||
## Usage
|
||||
|
||||
```
|
||||
/xce-test-resources [resource=value ...]
|
||||
run-resource-verify.sh [--keep-running] [--dir DIR] [resource=value ...]
|
||||
```
|
||||
|
||||
(the `/xce-test-resources [resource=value ...]` form above dispatches to this
|
||||
script.)
|
||||
|
||||
- No arguments: defaults to Requirement 1's gap-resource set — the 7 names
|
||||
`resupdate` is currently missing (`r15m`, `pg`, `ls`, `it`, `swp`, `io`,
|
||||
`tmp`) — plus a couple of already-supported names for regression coverage
|
||||
(e.g. `mem` for the numeric path, `type` for the string path), so a default
|
||||
run also proves the fix didn't disturb existing behavior.
|
||||
- One or more `resource=value` pairs: checks exactly those, e.g.
|
||||
`/xce-test-resources r15m=1.5 pg=3`.
|
||||
- Also accepts an optional scratch/deploy directory and a "keep running" flag
|
||||
(leaves the deployment up instead of cleaning up on exit) — exact flag/env
|
||||
names are pinned during implementation (tasks 5.1/7.1), following
|
||||
`xce-test-ha`'s `[rounds] [dir]` positional-argument convention and
|
||||
`XCE_HA_TOP`-style env vars as the closest precedent.
|
||||
`/xce-test-resources r15m=1.5 pg=3`. **Not yet implemented** — task 5.1
|
||||
parses and accepts these but does not act on them yet; injection lands in
|
||||
task 6.1.
|
||||
- `--dir DIR` — scratch/deploy directory. Default: a fresh directory
|
||||
generated via `mktemp -d` (xce-deploy-ha's "every invocation is a fresh,
|
||||
self-contained deploy" convention), or `$XCE_RESVERIFY_TOP` when set and
|
||||
`--dir` is not given (same precedence pattern as `xce-test-ha`'s
|
||||
`$XCE_HA_TOP`).
|
||||
- `--keep-running` — skip PROCESS teardown in cleanup, leaving the
|
||||
deployment running under `DIR` for further inspection. Log files under
|
||||
`DIR` are ALWAYS preserved regardless of this flag (see Error handling
|
||||
below); it only controls whether controller/scheduler/apiserver processes
|
||||
are killed.
|
||||
- Other overridable env vars (all optional, default to this repo's own
|
||||
meson build dir / `/tmp/apiserver`): `$XCE_RESVERIFY_BUILD_DIR`,
|
||||
`$XCE_RESVERIFY_BIN` (controller), `$XCE_RESVERIFY_SCHED_BIN`,
|
||||
`$XCE_RESVERIFY_APISERVER_BIN`, `$XCE_RESVERIFY_PEERS` (defaults to a
|
||||
single-node `localhost:9991` — this skill verifies value propagation, not
|
||||
raft/HA failover, so a single node is deliberate and avoids
|
||||
election-timing flakiness; xce-test-ha remains the tool for HA testing).
|
||||
|
||||
## VERDICT line and PASS/FAIL oracle
|
||||
|
||||
|
|
@ -170,15 +194,16 @@ or via the `Agent` tool
|
|||
(empirically verified safe for this exact pattern — see `xce-test-ha`'s and
|
||||
`xce-deploy-ha`'s own `SKILL.md`s), not via a forked Skill-tool context.
|
||||
|
||||
## Dependencies (not yet built)
|
||||
## Dependencies
|
||||
|
||||
- `xce_test`'s worker-context `resupdate`/`join` and client-context `resval`
|
||||
— existing, but `resupdate`'s resource-name table is currently incomplete
|
||||
(task 2.1 closes this).
|
||||
- `xce-deploy-ha`'s `deploy-ha.sh` / `start-all.sh` — controller deployment
|
||||
exists today; the apiserver step and combined `start-all.sh` are task 3.1.
|
||||
- Scheduler and apiserver trace-log lines — tasks 4.1 and 4.2.
|
||||
- This skill's own `run-resource-verify.sh` — tasks 5.1–7.1.
|
||||
— resource-name table gap closed (task 2.1/2.2, done).
|
||||
- `xce-deploy-ha`'s `deploy-ha.sh` / `start-all.sh` — apiserver step and
|
||||
combined `start-all.sh` done (task 3.1).
|
||||
- Scheduler and apiserver trace-log lines — done (tasks 4.1 and 4.2).
|
||||
- This skill's own `run-resource-verify.sh` — deployment step done (task
|
||||
5.1); injection + per-layer checks + verdict aggregation not yet built
|
||||
(tasks 6.1–7.1).
|
||||
|
||||
## Requirements traceability
|
||||
|
||||
|
|
|
|||
304
.claude/skills/xce-test-resources/run-resource-verify.sh
Executable file
304
.claude/skills/xce-test-resources/run-resource-verify.sh
Executable file
|
|
@ -0,0 +1,304 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# run-resource-verify.sh -- driver for /xce-test-resources: end-to-end
|
||||
# resource-value propagation verification (inject a synthetic worker
|
||||
# resource value via xce_test's resupdate, confirm it is visible at the
|
||||
# controller, live scheduler, and Go apiserver).
|
||||
#
|
||||
# resource-value-verification spec, task 5.1 SCOPE: this file currently
|
||||
# implements ONLY the deployment step -- stand up a fresh controller +
|
||||
# scheduler (-vvv) + apiserver (debug logging) deployment via
|
||||
# xce-deploy-ha's deploy-ha.sh + generated start-all.sh, confirm all three
|
||||
# components are actually up, and clean up (trap-based, covering a
|
||||
# mid-deployment abort, not just a completed run). Injection (task 6.1),
|
||||
# per-layer verification (tasks 6.2-6.4), and verdict aggregation /
|
||||
# cleanup-on-every-path (task 7.1) are added by LATER tasks -- see the
|
||||
# "Extension points" markers in main() below for exactly where they hook in.
|
||||
# See .kiro/specs/resource-value-verification/{requirements,design}.md,
|
||||
# Requirements 5.3, 6.1, 6.5 for this task; the full skill covers 3.1-6.5.
|
||||
#
|
||||
# Must be invoked under an environment with `nix` on PATH (same as
|
||||
# xce-deploy-ha / xce-test-ha) -- this script shells out to `nix develop`
|
||||
# itself for the steps that need it (openssl, xceadmin), so it does not need
|
||||
# to be pre-wrapped by the caller.
|
||||
#
|
||||
# Usage:
|
||||
# run-resource-verify.sh [--keep-running] [--dir DIR] [resource=value ...]
|
||||
#
|
||||
# --keep-running Skip PROCESS teardown in cleanup -- the deployment
|
||||
# (controller/scheduler/apiserver) is left running
|
||||
# under DIR for further inspection. Log files under
|
||||
# DIR are ALWAYS preserved regardless of this flag --
|
||||
# it only controls whether processes are killed (see
|
||||
# cleanup(), and design.md's explicit divergence from
|
||||
# xce-deploy-ha's own cleanup.sh, which wipes logs).
|
||||
# --dir DIR Scratch/deploy directory. Default: a fresh directory
|
||||
# generated via `mktemp -d`, following xce-deploy-ha's
|
||||
# "every invocation is a fresh, self-contained deploy"
|
||||
# convention. $XCE_RESVERIFY_TOP overrides the default
|
||||
# when --dir is not given (same precedence pattern as
|
||||
# xce-test-ha's $XCE_HA_TOP).
|
||||
# resource=value NOT YET IMPLEMENTED (task 6.1 -- worker join +
|
||||
# injection). Accepted and parsed here so argument
|
||||
# handling doesn't need reshaping later, but currently
|
||||
# unused by this task's deployment-only scope.
|
||||
#
|
||||
# Exit 0 == this task's scope succeeded (deployment stood up, all three
|
||||
# components confirmed up, cleaned up per --keep-running). Non-zero ==
|
||||
# deployment failed at some step (see the printed "deployment failed at
|
||||
# step X" message). Once tasks 6.1-7.1 land, the exit code instead reflects
|
||||
# the injection/verification PASS/FAIL verdict.
|
||||
#
|
||||
set -u
|
||||
|
||||
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO="$(cd "$SELF_DIR/../../.." && pwd)"
|
||||
|
||||
DEPLOY_HA="$REPO/src/tools/deploy-ha.sh"
|
||||
BUILD_DIR="${XCE_RESVERIFY_BUILD_DIR:-$REPO/src/xce.build}"
|
||||
BINARY="${XCE_RESVERIFY_BIN:-$BUILD_DIR/controller/xce_controller}"
|
||||
SCHED_BIN="${XCE_RESVERIFY_SCHED_BIN:-$BUILD_DIR/scheduler/xce_scheduler}"
|
||||
APISERVER_BIN="${XCE_RESVERIFY_APISERVER_BIN:-/tmp/apiserver}"
|
||||
|
||||
# Single-node cluster, deliberately -- this skill verifies resource-value
|
||||
# PROPAGATION (controller -> scheduler -> apiserver), not raft/HA failover
|
||||
# behavior (that's xce-test-ha's job). A single node still elects itself
|
||||
# CTRL_PRIMARY immediately and satisfies deploy-ha.sh's health-check, with
|
||||
# no election-timing flakiness to budget for. Override for multi-node via
|
||||
# $XCE_RESVERIFY_PEERS (comma-separated host:port list).
|
||||
PEERS="${XCE_RESVERIFY_PEERS:-localhost:9991}"
|
||||
|
||||
KEEP_RUNNING=0
|
||||
TOP=""
|
||||
RESOURCE_ARGS=() # task 6.1 extension point: resource=value pairs
|
||||
|
||||
log() { echo "[xce-test-resources] $*"; }
|
||||
fail() { echo "[xce-test-resources] FAIL: $*" >&2; exit 1; }
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $0 [--keep-running] [--dir DIR] [resource=value ...]
|
||||
|
||||
--keep-running leave controller/scheduler/apiserver processes running
|
||||
after this run instead of tearing them down. Logs are
|
||||
always preserved either way.
|
||||
--dir DIR scratch/deploy directory (default: fresh mktemp -d, or
|
||||
\$XCE_RESVERIFY_TOP).
|
||||
resource=value reserved for task 6.1 (injection); accepted but not yet
|
||||
acted on by this task's deployment-only scope.
|
||||
EOF
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--keep-running)
|
||||
KEEP_RUNNING=1
|
||||
shift
|
||||
;;
|
||||
--dir)
|
||||
[ $# -ge 2 ] || fail "--dir requires an argument"
|
||||
TOP="$2"
|
||||
shift 2
|
||||
;;
|
||||
--dir=*)
|
||||
TOP="${1#--dir=}"
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*=*)
|
||||
# resource=value pair -- task 6.1's concern; just collected here.
|
||||
RESOURCE_ARGS+=("$1")
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
echo "[xce-test-resources] Unknown argument: $1" >&2
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
resolve_top() {
|
||||
if [ -z "$TOP" ]; then
|
||||
TOP="${XCE_RESVERIFY_TOP:-$(mktemp -d /tmp/xce-resverify.XXXXXX)}"
|
||||
fi
|
||||
# Resolve a relative --dir/$XCE_RESVERIFY_TOP against cwd, so it is
|
||||
# unambiguous to deploy-ha.sh and every generated per-component script
|
||||
# (same pattern as xce-test-ha's run-ha-soak.sh; no realpath dependency
|
||||
# since busybox realpath in this environment lacks -m).
|
||||
case "$TOP" in
|
||||
/*) : ;;
|
||||
*) TOP="$PWD/$TOP" ;;
|
||||
esac
|
||||
mkdir -p "$TOP"
|
||||
}
|
||||
|
||||
# ── Cleanup: process teardown only, logs always preserved ──────────────────
|
||||
# Deliberately does NOT invoke xce-deploy-ha's own generated cleanup.sh --
|
||||
# that script wipes $TOP/logs/*.log, which this feature's design explicitly
|
||||
# diverges from (a FAIL verdict's underlying scheduler/apiserver trace-log
|
||||
# evidence must stay inspectable after the run). Every pkill pattern below is
|
||||
# scoped to a $TOP-unique argv substring, not a bare binary/process name --
|
||||
# xce-test-ha's bug-1436 postmortem: an unscoped `pkill -9 -x xce_controller`
|
||||
# can kill a DIFFERENT concurrent run's live cluster sharing the same
|
||||
# machine. $TOP itself (mktemp-unique per invocation, or an operator-chosen
|
||||
# --dir) is what makes each of these patterns safe to run concurrently with
|
||||
# an unrelated xce-deploy-ha/xce-test-ha/xce-test-resources invocation.
|
||||
cleanup() {
|
||||
if [ "$KEEP_RUNNING" -eq 1 ]; then
|
||||
log "--keep-running requested -- leaving controller/scheduler/apiserver up under $TOP (logs also preserved)"
|
||||
return 0
|
||||
fi
|
||||
log "cleaning up processes under $TOP (log files under $TOP/logs are preserved)"
|
||||
# One combined pattern (alternation), not four separate pkill
|
||||
# invocations -- fewer fork/exec round trips for a cleanup path that
|
||||
# may run from a signal trap, plus retry-until-verified below so a
|
||||
# slow/racy single kill attempt can't leave a process behind silently.
|
||||
local pattern="--workdir $TOP/controller| -l $TOP/logs|--logfile $TOP/logs/scheduler.log| -d $TOP/apiserver"
|
||||
local attempt=0
|
||||
while [ "$attempt" -lt 10 ]; do
|
||||
attempt=$((attempt + 1))
|
||||
if ! pgrep -f -- "$pattern" >/dev/null 2>&1; then
|
||||
break
|
||||
fi
|
||||
pkill -9 -f -- "$pattern" 2>/dev/null
|
||||
sleep 0.2
|
||||
done
|
||||
if pgrep -f -- "$pattern" >/dev/null 2>&1; then
|
||||
log "warning: some processes under $TOP may still be running after cleanup retries"
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# Standard robust trap idiom: on_exit (registered for EXIT) always runs
|
||||
# cleanup and preserves whatever exit code the script was already leaving
|
||||
# with. INT/TERM are trapped SEPARATELY and translated into an explicit
|
||||
# `exit`, because a bare `trap cleanup INT TERM` (with no exit inside the
|
||||
# handler) would only interrupt whatever foreground command was running and
|
||||
# then RESUME the script at the next statement -- silently defeating a
|
||||
# "kill the script mid-deployment" abort instead of actually stopping it.
|
||||
# Chaining through `exit N` here re-triggers the EXIT trap, so cleanup only
|
||||
# ever runs once.
|
||||
on_exit() {
|
||||
local rc=$?
|
||||
cleanup
|
||||
exit "$rc"
|
||||
}
|
||||
|
||||
register_trap() {
|
||||
trap on_exit EXIT
|
||||
trap 'exit 130' INT
|
||||
trap 'exit 143' TERM
|
||||
}
|
||||
|
||||
precondition_checks() {
|
||||
[ -x "$BINARY" ] || fail "controller binary missing: $BINARY (run /xce-build controller)"
|
||||
[ -x "$SCHED_BIN" ] || fail "scheduler binary missing: $SCHED_BIN (run /xce-build scheduler) -- required, this skill verifies the scheduler layer"
|
||||
[ -x "$APISERVER_BIN" ] || fail "apiserver binary missing: $APISERVER_BIN (cd src/frontend && make build-server, or the CGO-flags workaround -- see tasks.md's Implementation Notes) -- required, this skill verifies the apiserver layer"
|
||||
command -v nix >/dev/null 2>&1 || fail "nix not found on PATH -- required to resolve the deploy-environment bash and invoke deploy-ha.sh"
|
||||
}
|
||||
|
||||
# ── Step 1: deploy (xce-deploy-ha's deploy-ha.sh) ───────────────────────────
|
||||
deploy() {
|
||||
log "resolving deploy-environment bash via nix develop"
|
||||
BASH_BIN="$(nix develop "$REPO/src" --command which bash)"
|
||||
[ -n "$BASH_BIN" ] || fail "deployment failed at step deploy-ha.sh: could not resolve bash inside nix develop"
|
||||
|
||||
log "deploying controller+scheduler+apiserver into $TOP (peers=$PEERS)"
|
||||
if ! nix develop "$REPO/src" --command "$BASH_BIN" "$DEPLOY_HA" "$BINARY" "$TOP" "$PEERS"; then
|
||||
fail "deployment failed at step deploy-ha.sh (see output above)"
|
||||
fi
|
||||
|
||||
[ -x "$TOP/scripts/start-all.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start-all.sh was not generated"
|
||||
[ -x "$TOP/scripts/start-scheduler.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start-scheduler.sh was not generated (scheduler binary not found alongside controller build?)"
|
||||
[ -x "$TOP/scripts/start-apiserver.sh" ] || fail "deployment failed at step deploy-ha.sh: $TOP/scripts/start-apiserver.sh was not generated (no apiserver binary at $APISERVER_BIN?)"
|
||||
}
|
||||
|
||||
# ── Step 2: start-all.sh -- controller -> health-check -> scheduler/apiserver
|
||||
start_all() {
|
||||
log "invoking start-all.sh (controller, scheduler at -vvv, apiserver with debug logging)"
|
||||
if ! "$BASH_BIN" "$TOP/scripts/start-all.sh"; then
|
||||
fail "deployment failed at step start-all.sh (nonzero exit)"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Step 3: confirm all three components are actually up ───────────────────
|
||||
# start-all.sh's own internal health-check only gates on the CONTROLLER
|
||||
# electing a primary before it goes on to background scheduler/apiserver --
|
||||
# it does not itself wait for the scheduler to finish joining the cluster or
|
||||
# for the apiserver to finish registering with the controller before
|
||||
# returning. This skill's own "deployment succeeded" signal (distinct from
|
||||
# start-all.sh's) requires all three, so poll each with a short interval up
|
||||
# to a generous bounded wait (this codebase's established convention for
|
||||
# this kind of check -- see xce-test-ha and design.md's Implementation
|
||||
# Notes for the xce-test-resources skill).
|
||||
wait_for_log() {
|
||||
local desc="$1" pattern="$2" timeout="$3"
|
||||
shift 3
|
||||
local waited=0
|
||||
while [ "$waited" -lt "$timeout" ]; do
|
||||
if grep -qE -- "$pattern" "$@" 2>/dev/null; then
|
||||
log "confirmed: $desc"
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
waited=$((waited + 1))
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
confirm_deployment_up() {
|
||||
log "waiting for controller, scheduler, and apiserver to confirm up (bounded wait per component)"
|
||||
|
||||
wait_for_log "controller primary elected" "CTRL_PRIMARY" 30 $TOP/logs/ctrl-*.log \
|
||||
|| fail "deployment failed at step controller: no primary election observed in $TOP/logs/ctrl-*.log within 30s"
|
||||
|
||||
wait_for_log "scheduler joined cluster" "sched_init: joined cluster" 30 "$TOP/logs/scheduler.log" \
|
||||
|| fail "deployment failed at step scheduler: '$TOP/logs/scheduler.log' never logged joining the cluster within 30s"
|
||||
|
||||
wait_for_log "apiserver backend connection registered" "Backend connection registered" 30 "$TOP/logs/apiserver.log" \
|
||||
|| fail "deployment failed at step apiserver: '$TOP/logs/apiserver.log' never logged a registered backend connection within 30s"
|
||||
|
||||
log "all three components confirmed up under $TOP"
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
resolve_top
|
||||
|
||||
# Trap registered before the FIRST process is started (deploy() below
|
||||
# only generates scripts/certs/config; start_all() is what actually
|
||||
# starts backgrounded processes) -- this covers a mid-deployment abort
|
||||
# (e.g. apiserver fails to start after controller+scheduler are already
|
||||
# up), not just a completed PASS/FAIL run. See Requirement 6.5 and
|
||||
# design.md's Error Handling / Deployment failure entry.
|
||||
register_trap
|
||||
|
||||
precondition_checks
|
||||
deploy
|
||||
start_all
|
||||
confirm_deployment_up
|
||||
|
||||
# --- Extension point (task 6.1): join a synthetic worker via xce_test's
|
||||
# worker context and run resupdate for RESOURCE_ARGS (or the default
|
||||
# gap-resource set if RESOURCE_ARGS is empty) here. ---
|
||||
# --- Extension point (tasks 6.2-6.4): controller/scheduler/apiserver
|
||||
# per-layer verification checks here. ---
|
||||
# --- Extension point (task 7.1): aggregate results into a VERDICT line
|
||||
# plus the terminal PASS/FAIL oracle line, replacing this task's
|
||||
# placeholder success message and exit code below. ---
|
||||
|
||||
if [ "${#RESOURCE_ARGS[@]}" -gt 0 ]; then
|
||||
log "note: resource=value arguments (${RESOURCE_ARGS[*]}) were accepted but are not yet acted on -- injection lands in task 6.1"
|
||||
fi
|
||||
|
||||
log "PASS (task 5.1 scope): controller, scheduler, and apiserver confirmed up under $TOP; cleaning up"
|
||||
exit 0
|
||||
}
|
||||
|
||||
main "$@"
|
||||
|
|
@ -55,7 +55,7 @@
|
|||
- _Requirements: 5.1, 5.2, 5.4_
|
||||
|
||||
- [ ] 5. Integration: full-stack deployment orchestration
|
||||
- [ ] 5.1 Wire controller, scheduler, and apiserver deployment into the skill via `start-all.sh`
|
||||
- [x] 5.1 Wire controller, scheduler, and apiserver deployment into the skill via `start-all.sh`
|
||||
- Invoke `xce-deploy-ha`'s `deploy-ha.sh` (task 3.1) into a fresh scratch directory, then its generated `start-all.sh` — no manual per-component sequencing lives in this skill; `start-all.sh` already gates scheduler/apiserver startup on the controller health-check and handles the `-vvv`/debug-level logging needed for tasks 4.1/4.2's trace lines
|
||||
- Register a `trap`-based cleanup (or equivalent) before invoking `start-all.sh`, not just at the end of the script — this must cover a mid-deployment abort (e.g. apiserver fails to start after controller+scheduler are already up), not only a completed PASS/FAIL run; deliberately stronger than `xce-deploy-ha`'s own convention (a separate, manually-invoked `cleanup.sh`) because Requirement 6.5 requires cleanup-by-default on every path unless the operator passed the "keep running" flag
|
||||
- Observable completion: a single invocation of the orchestration script leaves a running controller, live scheduler daemon (built with task 4.1's trace log, started at `-vvv`), and apiserver (built with task 4.2's trace log, debug-level logging enabled), all logging under the same scratch directory, with no manual steps; killing the script mid-deployment (e.g. before the apiserver step completes) still removes every process it had already started, unless "keep running" was passed
|
||||
|
|
@ -116,6 +116,7 @@
|
|||
- _Depends: 7.1_
|
||||
|
||||
## Implementation Notes
|
||||
- Task 5.1: `run-resource-verify.sh` exists now (`.claude/skills/xce-test-resources/`), structured with explicit "Extension point" comments marking where task 6.1 (injection), 6.2-6.4 (per-layer checks), and 7.1 (verdict/cleanup-on-every-path) hook in — read those before starting 6.1. Trap is registered as the first statement in `main()`, before any process starts; `cleanup()` uses a single combined `pkill -9 -f` (alternation pattern) plus retry-until-`pgrep`-clean, not sequential per-process kills (a sequential version was tried first and found non-deterministically flaky under an async SIGTERM trap — bug-1575). Default deployment is single-node (not 3-node HA) — deliberate, since this skill verifies propagation not failover; override via `$XCE_RESVERIFY_PEERS`. Known non-blocking gap: `$XCE_RESVERIFY_SCHED_BIN`/`$XCE_RESVERIFY_APISERVER_BIN` only gate this script's own pre-flight check, NOT which binary `deploy-ha.sh` actually deploys (it hardcodes `/tmp/apiserver` and derives the scheduler path from `$BINARY`'s build dir) — don't rely on these two env vars to test against an alternate binary.
|
||||
- Task 4.2: the apiserver's trace log line (task 6.4 must grep for exactly this) is zap message `"resource-location stored"` with fields `resource` (string), `host` (string), `value` (float64 for numeric resources — the common case; string for `type`/`model`; bool for boolean-typed resources), e.g. `{"resource": "r15m", "host": "review-worker", "value": 1.75}`. Grep the log for `resource-location stored`, then parse `"resource": "<name>"` / `"host": "<host>"` / `"value": <val>` from the trailing field block.
|
||||
- **Task 4.2 discovered bug-1574, which will affect task 8.1 and possibly task 7.1's default resource set — read before implementing either**: `xce_test` can SEGFAULT on `resupdate -n type -v ...` / `resupdate -n model -v ...` (the two `Value_u_svalue`/string-typed resources). Root cause: `set_resval()`'s switch has no `Value_u_svalue` case, so `vals[idx].name` is left pointing at whatever garbage was already on `cmd_resupdate()`'s never-zero-initialized stack array — reading genuinely uninitialized stack memory, invisible to ASan (would need MemorySanitizer), and NON-DETERMINISTIC (task 2.2's reviewer saw `-n type -v foo` NOT crash under full ASAN; task 4.2's implementer saw `-n model -v CustomModelX` crash immediately — different argv/stack layout, same underlying UB). Task 1.2's `SKILL.md` scaffold already documents `type` as part of an illustrative default resource set "for string-path regression coverage" — **before task 7.1 finalizes the actual default resource set, either exclude `type`/`model` from any default until bug-1574 is fixed, or fix bug-1574 first** (it's a small, well-understood fix: zero-initialize `vals[]` in `cmd_resupdate()` and/or add the missing `Value_u_svalue` case to `set_resval()`'s value-population switch — out of boundary for tasks 4.1-4.2, in boundary for whichever task next touches `test.c`'s `cmd_resupdate()`/`set_resval()`, most likely as a prerequisite fix before task 8.1's live e2e run). A crash mid-run would make task 8.1's or the skill's own live verification flaky/non-deterministic, undermining the whole feature's purpose.
|
||||
- Consolidated list of pre-existing (not-fixed-here) bugs surfaced by this feature's own live testing so far: bug-1571 (apiserver `RegisterToBackend` gate checks `Address` before `Peers` — worked around in generated config, not source, by task 3.1), bug-1572 (scheduler double-free/crash on a resource's 2nd periodic report — task 4.1), bug-1573 (see buglog.json, task 4.1), bug-1574 (`xce_test` segfault on string-typed `resupdate` — task 4.2, described above).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue