feat(resource-value-verification): extend xce-deploy-ha with apiserver step + start-all.sh (task 3.1)
deploy-ha.sh Step 3 gains an opt-in-by-presence apiserver step (gated on a pre-built /tmp/apiserver binary, not built by this script) that generates start-apiserver.sh with a debug-level logger.yaml, plus an always-generated start-all.sh sequencing controller-health-check-then-worker/scheduler (-vvv)/apiserver. Purely additive - callers without a built apiserver (e.g. xce-test-ha) see zero behavior change. Live-confirmed both scenarios end-to-end: no-apiserver correctly skips the step; with-apiserver reaches full backend registration with debug logging active and scheduler emitting DEBUG3 output. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
fd5ab8b20d
commit
8f542a384a
SSH key fingerprint:
SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
3 changed files with 291 additions and 1 deletions
|
|
@ -37,6 +37,12 @@ disown`) before relying on subagent dispatch for unattended use.
|
|||
`cluster.bin` from `src/configs/*.yaml` fresh on every deploy
|
||||
- `openssl` must be on `PATH` (provided by `nix develop`) — `deploy-ha.sh`
|
||||
generates a fresh self-signed TLS cert/key for every deploy
|
||||
- Optional: run `cd src/frontend && make build-server` first if you want an
|
||||
apiserver step (Step 4d) — a separate Go build system from the meson build
|
||||
controller/worker/scheduler share, producing `/tmp/apiserver`.
|
||||
`deploy-ha.sh` never runs this itself; without it, Step 3 simply skips
|
||||
generating `scripts/start-apiserver.sh` (soft-skip, not an error), same as
|
||||
the existing worker/scheduler convention.
|
||||
|
||||
No data directory needed: TLS cert/key and `cluster.bin` are both generated
|
||||
directly into the deploy dir (`$TOP`) by `deploy-ha.sh` itself, not read from
|
||||
|
|
@ -145,6 +151,14 @@ This:
|
|||
`$TOP/scripts/xceadmin` **if** a wheel exists at
|
||||
`<same build dir>/xceadmin/dist/*.whl` — same soft-skip-with-warning
|
||||
convention. See Step 4c.
|
||||
- Also generates `$TOP/apiserver/config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml}`
|
||||
and `$TOP/scripts/start-apiserver.sh` **if** a pre-built apiserver binary
|
||||
exists at `/tmp/apiserver` (`make build-server`'s default output path —
|
||||
**not** alongside `$BINARY`, since the apiserver is a separate Go build,
|
||||
not part of this meson build dir) — same soft-skip-with-warning
|
||||
convention as worker/scheduler. See Step 4d.
|
||||
- Also **always** generates `$TOP/scripts/start-all.sh`, a combined shortcut
|
||||
for Steps 4/4b/4c — see below.
|
||||
|
||||
## Step 4 — Start all nodes
|
||||
|
||||
|
|
@ -217,6 +231,67 @@ a warning if no wheel exists in the same build dir as `$BINARY`, same as
|
|||
`--controller` silently tried port 9991 and failed with a connection-
|
||||
refused that had nothing to do with the actual deployment.
|
||||
|
||||
## Step 4d — Start the apiserver (optional)
|
||||
|
||||
`deploy-ha.sh` also generates `$TOP/scripts/start-apiserver.sh` **if** a
|
||||
pre-built apiserver binary exists at `/tmp/apiserver` (soft-skipped with a
|
||||
warning, not a hard error, if missing — see Prerequisites). Start it once at
|
||||
least one controller is up, the same way as the worker/scheduler:
|
||||
|
||||
```bash
|
||||
"$BASH_BIN" "$TOP/scripts/start-apiserver.sh" &
|
||||
```
|
||||
|
||||
- Runs the apiserver binary with `-d "$TOP/apiserver"` as its top directory,
|
||||
pointing it at `$TOP/apiserver/config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml}`
|
||||
— a **minimal boot-only config**, not a production one:
|
||||
- `auth.yaml`/`rbac.yaml` are copied verbatim from the repo's own shipped
|
||||
dev-mode templates (`src/frontend/apiserver/config/`) — functional
|
||||
auth/RBAC is not required for this deployment (no REST query is ever
|
||||
issued against it here), only enough to load without erroring.
|
||||
- `app.yaml` is copied from the same shipped template, with its
|
||||
`backend:` section replaced to point `peers:` at this deployment's own
|
||||
`ctrl-N=host:port` candidate list (the same one every other generated
|
||||
script here already uses) — without this, the apiserver never connects
|
||||
to the controller at all.
|
||||
- `logger.yaml` is generated fresh (not copied) with `logging.level:
|
||||
debug` explicitly set — required for any future apiserver-side
|
||||
trace-log observability to actually be emitted; at any less-verbose
|
||||
level it is silently dropped.
|
||||
- Logs to `$TOP/logs/apiserver.log` — same shared `logs/` directory
|
||||
`ctrl-N.log`/`scheduler.log` already use (an absolute `outputPath` in
|
||||
`logger.yaml`, even though the apiserver's own topDir is the separate
|
||||
`$TOP/apiserver`).
|
||||
- Verify it started and connected to the controller:
|
||||
`grep "Backend connection registered" "$TOP/logs/apiserver.log"`.
|
||||
- Does not attempt HA/multi-instance apiserver deployment — single instance
|
||||
only.
|
||||
|
||||
## Step 4e — `start-all.sh`: Steps 4/4b/4c/4d in one shot
|
||||
|
||||
`deploy-ha.sh` **always** generates `$TOP/scripts/start-all.sh`, a combined
|
||||
shortcut that automates the manual recipe above end to end:
|
||||
|
||||
```bash
|
||||
"$BASH_BIN" "$TOP/scripts/start-all.sh"
|
||||
```
|
||||
|
||||
It: starts every controller node (`start.sh ctrl-N -vvv`, staggered 0.5s
|
||||
apart, same as Step 4); waits up to 30s for the same primary-election
|
||||
health-check Step 5 checks for; then starts whichever of
|
||||
`start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually
|
||||
generated for this deploy — checked via `[ -x ... ]` at `start-all.sh`'s own
|
||||
runtime, so it correctly skips any step whose binary wasn't built, with no
|
||||
regeneration needed. The scheduler is started at `-vvv` specifically (not
|
||||
just `-v`) so its `TRACE`/`debug3`-level output is reliably emitted. Every
|
||||
process is backgrounded via `setsid nohup … </dev/null >/dev/null 2>&1 &
|
||||
disown`, so the whole cluster survives `start-all.sh`'s own exit — safe to
|
||||
run from a subagent dispatch the same way Steps 1–5 already are (see the
|
||||
note at the top of this file).
|
||||
|
||||
`start-all.sh` is regenerated fresh on every Step 3 deploy, same as every
|
||||
other generated script — nothing to keep in sync by hand.
|
||||
|
||||
## Step 5 — Verify cluster health
|
||||
|
||||
Wait ~15 seconds for election and config replication, then check:
|
||||
|
|
@ -303,3 +378,7 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins
|
|||
| A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy |
|
||||
| `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER="<first --peers candidate>"` before the `exec` line |
|
||||
| `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) |
|
||||
| `scripts/start-apiserver.sh` not generated | No binary at `/tmp/apiserver` (`deploy-ha.sh` logs a `Warning:`, not an error) — this is a separate Go build, not part of `$BINARY`'s meson build dir | `cd src/frontend && make build-server`, then redeploy (Step 3) |
|
||||
| Apiserver exits immediately / `Failed to initialize configuration files` | `$TOP/apiserver/config/app.yaml` failed to parse or is missing a required field (e.g. `file_upload.max_size`) — most likely the shipped `src/frontend/apiserver/config/app.yaml` template this is copied from changed shape | Inspect `$TOP/logs/apiserver.log` / `$TOP/apiserver/logs/logger_error.log`; diff `$TOP/apiserver/config/app.yaml` against the shipped template |
|
||||
| Apiserver boots but never logs `Backend connection registered` | `app.yaml`'s `backend.peers` doesn't match the controller cluster (stale redeploy), or no controller is up yet | Confirm `$TOP/apiserver/config/app.yaml`'s `backend.peers` matches `$TOP/scripts/start.sh`'s `--peers`; start (or wait for) a controller first |
|
||||
| `start-all.sh` reports "no primary election observed... proceeding anyway" | Same causes as `election_timeout` above — no controller became primary within 30s | Check controller logs directly (Step 5); worker/scheduler/apiserver may still fail to connect until a primary exists |
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@
|
|||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 3. Core: apiserver local deployment capability
|
||||
- [ ] 3.1 (P) Extend `xce-deploy-ha`'s `deploy-ha.sh` with a conditional apiserver step and a combined `start-all.sh`
|
||||
- [x] 3.1 (P) Extend `xce-deploy-ha`'s `deploy-ha.sh` with a conditional apiserver step and a combined `start-all.sh`
|
||||
- In `deploy-ha.sh` Step 3, add a new conditional branch in the same *opt-in-by-presence* spirit as the existing worker/scheduler branch: check for a pre-built apiserver binary at its own Go-build output path (e.g. `/tmp/apiserver`, `make build-server`'s default) — NOT alongside `$BINARY`, since apiserver is a separate build system, not the same meson build dir worker/scheduler share; `deploy-ha.sh` does not invoke `make build-server` itself, building it remains a caller prerequisite (document in SKILL.md's Prerequisites section, same pattern as the existing `/xce-build controller` note). If present, generate `$TOP/scripts/start-apiserver.sh` plus a minimal `logger.yaml` (modeled on `make test-integration`'s `XCE_BIN`/`SRV_URL` convention) — functional auth/RBAC is NOT required (verification is trace-log-based, task 6.4; no REST query is ever issued against this instance), but `logger.yaml` MUST set `logging.level: debug` or task 4.2's trace line is silently never written; absent the binary, generate nothing, so existing callers are unaffected
|
||||
- Also in Step 3, always generate `$TOP/scripts/start-all.sh`: runs `start.sh` for the controller node(s), waits on Step 5's existing health-check, then runs whichever of `start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually generated for this deploy (skipping any that weren't) — start the scheduler with `-vvv` specifically (verbosity 3, matching `tlog_debug3`/`TRACE`), overriding SKILL.md's existing `-v`-only example for this feature's needs, or task 4.1's trace line is silently never written
|
||||
- Update `.claude/skills/xce-deploy-ha/SKILL.md` with a new "Step 4d" documenting the apiserver step (following the existing Step 4b pattern) and documenting `start-all.sh` as a combined shortcut for Steps 4/4b/4c
|
||||
|
|
@ -118,5 +118,6 @@
|
|||
## Implementation Notes
|
||||
- Task 1.1: `resval_typeresolve.c`'s new function is `int resval_resolve_type(const char* name, int type_override)`, returning a `Value_u_which` tag or `RESVAL_TYPE_UNKNOWN` (-1, doesn't collide with the real 0-6 enum range). Task 2.1 should build on this exact signature rather than inventing a new one. `test.c`'s `set_resval()`/`cmd_resupdate()` were deliberately left untouched by task 1.1 — task 2.1 is the one that makes `set_resval()` a thin wrapper.
|
||||
- `controller_unit_test` (the big legacy Unity binary) has a pre-existing, unrelated flaky failure in `test_config_repl.c` (bug-1561, overlayfs `fdatasync errno=5` sandbox artifact) that shows up on every `/xce-build controller` run in this environment. It is NOT caused by this feature's changes — do not chase it in later tasks; cross-check `.wolf/buglog.json`'s bug-1561 entry before treating a `controller_unit_test` failure as real, and confirm via `git status` that `test_config_repl.c`/`config_repl.c` are untouched by the current task's diff.
|
||||
- Task 3.1: `src/frontend/Makefile`'s `build-server` target does not work standalone in this environment — it never sets `CGO_CFLAGS`/`CGO_LDFLAGS` the way `test-backend` does, failing with `fatal error: client.h: No such file or directory`. Pre-existing, out-of-boundary for this feature (confirmed unrelated to any of this feature's changes). To build a real apiserver binary for a live smoke test, export the same CGO flags `make test-backend` derives before running `go build`. Worth a separate bug-log entry / follow-up fix to the Makefile itself, not fixed here. Also found and worked around (config-only, no source change): `main.go`'s `RegisterToBackend` gate checks `BackendYamlConfig.Address` before consulting `Peers`, so a generated config needs `address`/`port` populated from the first `--peers` candidate, not just `peers` itself (bug-1571).
|
||||
- Task 2.2: `src/controller/meson.build` compiles `test.c` only into the `xce_test` binary (`xce_test_src`), never into `controller_unit_test` (`controller_unit_test_src`) — a `test.c`-only diff structurally cannot regress `controller_unit_test`'s results. Also: `controller_unit_test` is built with ASAN `detect_leaks=0` as pre-existing, repo-wide meson policy (not something any task in this feature turned off) — to actually check a `test.c` change for new leaks, run `xce_test` directly with leak detection left ON, don't rely on the meson suite's leak-detection status.
|
||||
- Task 2.1 (round 1 REJECTED, round 2 fixed): refactoring a getopt-loop's flush/finalization timing (needed to let a trailing flag like `-t` apply to its preceding `-n`/`-v` pair) can silently drop an unrelated existing guard (here: duplicate-`-v` rejection) that depended on the old immediate-finalization timing. When touching `cmd_resupdate()`-shaped CLI parsers in this codebase again, explicitly re-trace EVERY branch of the old getopt loop against the new one, not just the branches directly related to the new flag — a "traced every path" claim is only as good as the list of paths actually enumerated. `xce_test` also needs `ASAN_OPTIONS=detect_leaks=0` for reliable non-interactive stdin-piped smoke testing (its default leak-detection exit path can swallow buffered stdout).
|
||||
|
|
|
|||
|
|
@ -96,6 +96,27 @@ if [ ! -x "$SCHED_BIN" ]; then
|
|||
SCHED_BIN=""
|
||||
fi
|
||||
|
||||
# Apiserver binary check -- same opt-in-by-presence spirit as WORKER_BIN/
|
||||
# SCHED_BIN above, but NOT a same-build-generation sibling lookup: the
|
||||
# apiserver is a SEPARATE build system (Go, `make build-server` in
|
||||
# src/frontend/Makefile), not part of the meson build dir $BINARY lives in,
|
||||
# so there is no "$BINARY_BUILD_DIR/apiserver/..." to check. Its own build
|
||||
# output always lands at a fixed path (`/tmp/apiserver`, build-server's
|
||||
# `go build -o` target) regardless of which meson profile/build dir $BINARY
|
||||
# came from. deploy-ha.sh does NOT invoke `make build-server` itself --
|
||||
# building it remains a caller prerequisite (same "run /xce-build controller
|
||||
# first if stale" convention SKILL.md's Prerequisites section already
|
||||
# documents for the controller, just a different build command). Soft-check
|
||||
# only, same reasoning as worker/scheduler: xce-test-ha never builds an
|
||||
# apiserver and calls this same script, so a missing binary must not fail
|
||||
# the deploy -- it just skips generating scripts/start-apiserver.sh, with a
|
||||
# warning.
|
||||
APISERVER_BIN="/tmp/apiserver"
|
||||
if [ ! -x "$APISERVER_BIN" ]; then
|
||||
echo "Warning: no apiserver binary found at $APISERVER_BIN -- skipping scripts/start-apiserver.sh generation (run 'cd src/frontend && make build-server' first if you need it)"
|
||||
APISERVER_BIN=""
|
||||
fi
|
||||
|
||||
# xceadmin wheel (same-build-generation sibling again): /xce-build xceadmin
|
||||
# leaves one at <builddir>/xceadmin/dist/xceadmin-*.whl (meson-build.sh).
|
||||
# Installed below into its OWN venv under $TOP, not just pointed at
|
||||
|
|
@ -241,6 +262,119 @@ EOF
|
|||
chmod +x "$TOP/scripts/start-scheduler.sh"
|
||||
fi
|
||||
|
||||
# Generate apiserver config + start script -- only if a pre-built apiserver
|
||||
# binary was found above. This is NOT an exact mirror of the worker/
|
||||
# scheduler conditional: those two just launch an existing meson-build
|
||||
# binary with --peers; the apiserver additionally needs its own topDir
|
||||
# (`-d`) carrying config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml} before
|
||||
# it will even boot (utils.PromptForTopDir/InitConf/InitLogger all read
|
||||
# from "$TOPDIR/config/*.yaml"), so this block also generates that config,
|
||||
# not just a launcher script.
|
||||
#
|
||||
# The apiserver's own topDir is $TOP/apiserver (its own config/ subdir,
|
||||
# separate from the controller's $TOP/controller/ctrl-N/conf -- they must
|
||||
# not collide). Its log file, however, is pointed via logger.yaml's
|
||||
# outputPath at an ABSOLUTE path directly under $TOP/logs -- logger.go only
|
||||
# prepends "<topdir>/logs/" to a RELATIVE outputPath, so an absolute one
|
||||
# here lands the apiserver's log in the SAME shared logs/ directory
|
||||
# ctrl-N.log/scheduler.log already use, matching this deploy's existing
|
||||
# logs/ convention (this is also where a later verification skill greps
|
||||
# for the apiserver's resource-update trace line).
|
||||
if [ -n "$APISERVER_BIN" ]; then
|
||||
APISERVER_SRC_CONFIG="$SCRIPT_DIR/../frontend/apiserver/config"
|
||||
mkdir -p "$TOP/apiserver/config"
|
||||
|
||||
# auth.yaml / rbac.yaml: copied verbatim from the repo's own shipped
|
||||
# dev-mode templates (config_file provider, built-in admin/basic_user
|
||||
# roles) -- functional auth/RBAC is NOT required for this deploy (no
|
||||
# REST query is ever issued against this apiserver instance), only
|
||||
# enough for auth.InitAuthProvider()/controllers.LoadRBACConfig() to
|
||||
# load successfully at boot without erroring.
|
||||
cp "$APISERVER_SRC_CONFIG/auth.yaml" "$TOP/apiserver/config/auth.yaml"
|
||||
cp "$APISERVER_SRC_CONFIG/rbac.yaml" "$TOP/apiserver/config/rbac.yaml"
|
||||
|
||||
# app.yaml: copied verbatim from the shipped template (keeps every
|
||||
# required field -- e.g. file_upload.max_size, which InitConf()
|
||||
# strictly validates and errors out on if missing/malformed -- in sync
|
||||
# with whatever the repo's own default currently is), then its
|
||||
# trailing `backend:` section (the last top-level section in the
|
||||
# shipped file today) is stripped and replaced with one pointing at
|
||||
# THIS deploy's own controller cluster: the identical
|
||||
# ctrl-N=host:port candidate list every other generated script here
|
||||
# already uses, in the exact "[name=]host:port" syntax
|
||||
# RegisterToBackend's xce_parse_peers expects (backend.peers takes
|
||||
# priority over the legacy single address/port fallback -- see
|
||||
# apiserver/backend/config.go). Without this, BackendYamlConfig.Address
|
||||
# stays empty and main.go never calls RegisterToBackend at all --
|
||||
# the apiserver would boot in permanent stub mode and
|
||||
# OnResourceLocationUpdated would never fire.
|
||||
sed '/^backend:/,$d' "$APISERVER_SRC_CONFIG/app.yaml" > "$TOP/apiserver/config/app.yaml"
|
||||
APISERVER_PEERS_YAML=""
|
||||
for i in "${!PEER_ARRAY[@]}"; do
|
||||
n=$((i+1))
|
||||
APISERVER_PEERS_YAML+=" - \"ctrl-$n=${PEER_ARRAY[$i]}\"
|
||||
"
|
||||
done
|
||||
# address/port are still populated (first --peers candidate), even
|
||||
# though `peers` (non-empty) is what RegisterToBackend actually uses to
|
||||
# pick a connection candidate -- confirmed live: main.go's own gate for
|
||||
# whether to call RegisterToBackend AT ALL is `BackendYamlConfig.Address
|
||||
# != ""`, checked BEFORE Peers is ever consulted (cmd/main.go:58). A
|
||||
# blank address here (even with a fully-populated non-empty peers list)
|
||||
# left the apiserver silently booting in permanent stub mode ("Backend
|
||||
# not configured") with RegisterToBackend never called -- caught during
|
||||
# this task's own live verification, not a hypothetical.
|
||||
APISERVER_FIRST_HOST="${PEER_ARRAY[0]%%:*}"
|
||||
APISERVER_FIRST_PORT="${PEER_ARRAY[0]##*:}"
|
||||
cat >> "$TOP/apiserver/config/app.yaml" << EOF
|
||||
backend:
|
||||
address: "$APISERVER_FIRST_HOST"
|
||||
port: $APISERVER_FIRST_PORT
|
||||
from: "xce_apiserver"
|
||||
key: "keepsecret\$"
|
||||
peers:
|
||||
$APISERVER_PEERS_YAML
|
||||
EOF
|
||||
|
||||
# logger.yaml: NOT copied from the shipped template -- generated fresh
|
||||
# so `logging.level: debug` is explicit and load-bearing here, rather
|
||||
# than incidentally inherited from whatever the shipped template's
|
||||
# default happens to be today. A later trace-log addition at the
|
||||
# apiserver's resource-update storage point is only ever emitted via
|
||||
# logger.Debug(...) (zap); at any less-verbose configured level it is
|
||||
# silently dropped, which would look identical to a genuine
|
||||
# propagation failure to anything polling this log.
|
||||
cat > "$TOP/apiserver/config/logger.yaml" << EOF
|
||||
logging:
|
||||
level: "debug"
|
||||
encoding: console
|
||||
outputPath: "$TOP/logs/apiserver.log"
|
||||
file:
|
||||
maxSize: 100
|
||||
maxBackups: 10
|
||||
maxAge: 14
|
||||
compress: true
|
||||
encoderConfig:
|
||||
messageKey: "message"
|
||||
levelKey: "level"
|
||||
timeKey: "timestamp"
|
||||
nameKey: "logger"
|
||||
callerKey: "caller"
|
||||
functionKey: "func"
|
||||
stacktraceKey: "stacktrace"
|
||||
timeEncoder: "ISO8601"
|
||||
levelEncoder: "capital"
|
||||
EOF
|
||||
|
||||
cat > "$TOP/scripts/start-apiserver.sh" << EOF
|
||||
#!$GEN_BASH
|
||||
set -e
|
||||
mkdir -p $TOP/logs
|
||||
$APISERVER_BIN -d $TOP/apiserver \$*
|
||||
EOF
|
||||
chmod +x "$TOP/scripts/start-apiserver.sh"
|
||||
fi
|
||||
|
||||
# Install xceadmin into THIS deployment ($TOP/xceadmin-site), not just
|
||||
# pointed at $XCEADMIN's build-dir .venv resolved above -- a real, standalone
|
||||
# copy, so the deployment keeps a working xceadmin long after the build dir
|
||||
|
|
@ -297,8 +431,84 @@ EOF
|
|||
chmod +x "$TOP/scripts/xceadmin"
|
||||
fi
|
||||
|
||||
# Generate start-all.sh -- ALWAYS (unlike start-worker.sh/start-scheduler.sh/
|
||||
# start-apiserver.sh above, this one is not conditional on any optional
|
||||
# binary's presence). It automates the exact manual recipe SKILL.md's Steps
|
||||
# 4/4b/4c already document a human running by hand: start every controller
|
||||
# node, wait for the same primary-election health-check Step 5 documents,
|
||||
# then start whichever of start-worker.sh/start-scheduler.sh/
|
||||
# start-apiserver.sh were actually generated for THIS deploy -- checked at
|
||||
# the generated script's OWN runtime via [ -x ... ], not baked in as a
|
||||
# deploy-time boolean, so start-all.sh is a single self-contained artifact
|
||||
# that correctly no-ops a step whose binary wasn't available, with no
|
||||
# regeneration needed if e.g. a stale copy of start-all.sh is run after
|
||||
# manually deleting one of the optional scripts.
|
||||
#
|
||||
# Every component is started via `setsid nohup ... </dev/null >/dev/null
|
||||
# 2>&1 & disown`, not a plain `&` -- reparents each process to init so it
|
||||
# survives start-all.sh's own exit (needed since this script is meant to be
|
||||
# run once and return, leaving the cluster running for later interactive
|
||||
# use, exactly like this skill's own SKILL.md already recommends for
|
||||
# Agent-subagent dispatch of Steps 1-5 and xce-test-ha's soak driver).
|
||||
#
|
||||
# The scheduler is started with -vvv specifically (not the -v this file's
|
||||
# own start-scheduler.sh body already bakes in unconditionally) so its
|
||||
# resulting verbosity is >= 3 (`debug3`) -- verbose_to_loglevel() caps at
|
||||
# "debug3" for any count above 3, so stacking with the wrapper's own -v is
|
||||
# harmless, but passing anything less than vvv here risks the trace log a
|
||||
# later resource-verification feature depends on being silently never
|
||||
# written at the daemon's default verbosity.
|
||||
cat > "$TOP/scripts/start-all.sh" << EOF
|
||||
#!$GEN_BASH
|
||||
set -e
|
||||
|
||||
echo "=== start-all.sh: starting controller node(s) ==="
|
||||
for i in \$(seq 1 $N); do
|
||||
setsid nohup "$GEN_BASH" "$TOP/scripts/start.sh" ctrl-\$i -vvv </dev/null >/dev/null 2>&1 &
|
||||
disown
|
||||
sleep 0.5
|
||||
done
|
||||
|
||||
echo "=== start-all.sh: waiting for controller health-check (up to 30s) ==="
|
||||
HEALTHY=0
|
||||
for i in \$(seq 1 30); do
|
||||
if grep -lE "CTRL_PRIMARY" $TOP/logs/ctrl-*.log >/dev/null 2>&1; then
|
||||
HEALTHY=1
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
if [ "\$HEALTHY" -eq 0 ]; then
|
||||
echo "Warning: no primary election observed in $TOP/logs/ctrl-*.log within 30s -- proceeding anyway (worker/scheduler/apiserver may fail to connect)"
|
||||
else
|
||||
echo "Controller cluster healthy (primary elected)."
|
||||
fi
|
||||
|
||||
if [ -x "$TOP/scripts/start-worker.sh" ]; then
|
||||
echo "=== start-all.sh: starting worker ==="
|
||||
setsid nohup "$GEN_BASH" "$TOP/scripts/start-worker.sh" worker-1 -vvv </dev/null >/dev/null 2>&1 &
|
||||
disown
|
||||
fi
|
||||
|
||||
if [ -x "$TOP/scripts/start-scheduler.sh" ]; then
|
||||
echo "=== start-all.sh: starting scheduler (-vvv) ==="
|
||||
setsid nohup "$GEN_BASH" "$TOP/scripts/start-scheduler.sh" -vvv </dev/null >/dev/null 2>&1 &
|
||||
disown
|
||||
fi
|
||||
|
||||
if [ -x "$TOP/scripts/start-apiserver.sh" ]; then
|
||||
echo "=== start-all.sh: starting apiserver ==="
|
||||
setsid nohup "$GEN_BASH" "$TOP/scripts/start-apiserver.sh" </dev/null >/dev/null 2>&1 &
|
||||
disown
|
||||
fi
|
||||
|
||||
echo "=== start-all.sh: done -- cluster left running ==="
|
||||
EOF
|
||||
chmod +x "$TOP/scripts/start-all.sh"
|
||||
|
||||
echo "Deployed $N-node cluster to $TOP (tag: $TAG)"
|
||||
if [ -n "$XCEADMIN_WHEEL" ]; then
|
||||
echo "xceadmin installed at $TOP/xceadmin-site -- use it via $TOP/scripts/xceadmin, e.g.:"
|
||||
echo " $TOP/scripts/xceadmin config push $TOP/controller/ctrl-1/conf/cluster.bin.$TAG --controller ${PEER_ARRAY[0]}"
|
||||
fi
|
||||
echo "Run \"$GEN_BASH\" $TOP/scripts/start-all.sh to start controller(s) plus whichever of worker/scheduler/apiserver were generated for this deploy."
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue