From 8f542a384a9fd1cb451f00175552ce87c4e1e9ac Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Tue, 18 Aug 2026 00:53:49 +0000 Subject: [PATCH] feat(resource-value-verification): extend xce-deploy-ha with apiserver step + start-all.sh (task 3.1) deploy-ha.sh Step 3 gains an opt-in-by-presence apiserver step (gated on a pre-built /tmp/apiserver binary, not built by this script) that generates start-apiserver.sh with a debug-level logger.yaml, plus an always-generated start-all.sh sequencing controller-health-check-then-worker/scheduler (-vvv)/apiserver. Purely additive - callers without a built apiserver (e.g. xce-test-ha) see zero behavior change. Live-confirmed both scenarios end-to-end: no-apiserver correctly skips the step; with-apiserver reaches full backend registration with debug logging active and scheduler emitting DEBUG3 output. Co-Authored-By: Claude Sonnet 5 --- .claude/skills/xce-deploy-ha/SKILL.md | 79 +++++++ .../resource-value-verification/tasks.md | 3 +- src/tools/deploy-ha.sh | 210 ++++++++++++++++++ 3 files changed, 291 insertions(+), 1 deletion(-) diff --git a/.claude/skills/xce-deploy-ha/SKILL.md b/.claude/skills/xce-deploy-ha/SKILL.md index 49d2e73a..5b610212 100644 --- a/.claude/skills/xce-deploy-ha/SKILL.md +++ b/.claude/skills/xce-deploy-ha/SKILL.md @@ -37,6 +37,12 @@ disown`) before relying on subagent dispatch for unattended use. `cluster.bin` from `src/configs/*.yaml` fresh on every deploy - `openssl` must be on `PATH` (provided by `nix develop`) — `deploy-ha.sh` generates a fresh self-signed TLS cert/key for every deploy +- Optional: run `cd src/frontend && make build-server` first if you want an + apiserver step (Step 4d) — a separate Go build system from the meson build + controller/worker/scheduler share, producing `/tmp/apiserver`. + `deploy-ha.sh` never runs this itself; without it, Step 3 simply skips + generating `scripts/start-apiserver.sh` (soft-skip, not an error), same as + the existing worker/scheduler convention. No data directory needed: TLS cert/key and `cluster.bin` are both generated directly into the deploy dir (`$TOP`) by `deploy-ha.sh` itself, not read from @@ -145,6 +151,14 @@ This: `$TOP/scripts/xceadmin` **if** a wheel exists at `/xceadmin/dist/*.whl` — same soft-skip-with-warning convention. See Step 4c. +- Also generates `$TOP/apiserver/config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml}` + and `$TOP/scripts/start-apiserver.sh` **if** a pre-built apiserver binary + exists at `/tmp/apiserver` (`make build-server`'s default output path — + **not** alongside `$BINARY`, since the apiserver is a separate Go build, + not part of this meson build dir) — same soft-skip-with-warning + convention as worker/scheduler. See Step 4d. +- Also **always** generates `$TOP/scripts/start-all.sh`, a combined shortcut + for Steps 4/4b/4c — see below. ## Step 4 — Start all nodes @@ -217,6 +231,67 @@ a warning if no wheel exists in the same build dir as `$BINARY`, same as `--controller` silently tried port 9991 and failed with a connection- refused that had nothing to do with the actual deployment. +## Step 4d — Start the apiserver (optional) + +`deploy-ha.sh` also generates `$TOP/scripts/start-apiserver.sh` **if** a +pre-built apiserver binary exists at `/tmp/apiserver` (soft-skipped with a +warning, not a hard error, if missing — see Prerequisites). Start it once at +least one controller is up, the same way as the worker/scheduler: + +```bash +"$BASH_BIN" "$TOP/scripts/start-apiserver.sh" & +``` + +- Runs the apiserver binary with `-d "$TOP/apiserver"` as its top directory, + pointing it at `$TOP/apiserver/config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml}` + — a **minimal boot-only config**, not a production one: + - `auth.yaml`/`rbac.yaml` are copied verbatim from the repo's own shipped + dev-mode templates (`src/frontend/apiserver/config/`) — functional + auth/RBAC is not required for this deployment (no REST query is ever + issued against it here), only enough to load without erroring. + - `app.yaml` is copied from the same shipped template, with its + `backend:` section replaced to point `peers:` at this deployment's own + `ctrl-N=host:port` candidate list (the same one every other generated + script here already uses) — without this, the apiserver never connects + to the controller at all. + - `logger.yaml` is generated fresh (not copied) with `logging.level: + debug` explicitly set — required for any future apiserver-side + trace-log observability to actually be emitted; at any less-verbose + level it is silently dropped. +- Logs to `$TOP/logs/apiserver.log` — same shared `logs/` directory + `ctrl-N.log`/`scheduler.log` already use (an absolute `outputPath` in + `logger.yaml`, even though the apiserver's own topDir is the separate + `$TOP/apiserver`). +- Verify it started and connected to the controller: + `grep "Backend connection registered" "$TOP/logs/apiserver.log"`. +- Does not attempt HA/multi-instance apiserver deployment — single instance + only. + +## Step 4e — `start-all.sh`: Steps 4/4b/4c/4d in one shot + +`deploy-ha.sh` **always** generates `$TOP/scripts/start-all.sh`, a combined +shortcut that automates the manual recipe above end to end: + +```bash +"$BASH_BIN" "$TOP/scripts/start-all.sh" +``` + +It: starts every controller node (`start.sh ctrl-N -vvv`, staggered 0.5s +apart, same as Step 4); waits up to 30s for the same primary-election +health-check Step 5 checks for; then starts whichever of +`start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually +generated for this deploy — checked via `[ -x ... ]` at `start-all.sh`'s own +runtime, so it correctly skips any step whose binary wasn't built, with no +regeneration needed. The scheduler is started at `-vvv` specifically (not +just `-v`) so its `TRACE`/`debug3`-level output is reliably emitted. Every +process is backgrounded via `setsid nohup … /dev/null 2>&1 & +disown`, so the whole cluster survives `start-all.sh`'s own exit — safe to +run from a subagent dispatch the same way Steps 1–5 already are (see the +note at the top of this file). + +`start-all.sh` is regenerated fresh on every Step 3 deploy, same as every +other generated script — nothing to keep in sync by hand. + ## Step 5 — Verify cluster health Wait ~15 seconds for election and config replication, then check: @@ -303,3 +378,7 @@ If you need a full redeploy (different peers or new binary), run steps 3–5 ins | A generated script fails with `bad interpreter` or silently uses the wrong bash | Redeploying with a stale `$BASH_BIN` captured outside `nix develop`, or a shell that shadows `bash` | Re-resolve `BASH_BIN=$(nix develop ./src --command which bash)` right before Step 3 and re-deploy | | `scripts/xceadmin ...` (no `--controller`) fails `cannot connect to localhost:9991: Connection refused` | bug-1566, fixed — a wheel built before this fix, or a `deploy-ha.sh` from before it, doesn't export `XCEADMIN_CONTROLLER` | Rebuild (`/xce-build xceadmin --incremental`) and redeploy (Step 3); check `$TOP/scripts/xceadmin` — it should `export XCEADMIN_CONTROLLER=""` before the `exec` line | | `scripts/xceadmin` not generated | No wheel in `$BINARY`'s build dir (`deploy-ha.sh` logs a `Warning:`, not an error) | `/xce-build xceadmin` (add `--incremental` if you don't want to lose other already-built binaries in that dir — see `/xce-build`'s SKILL.md), then redeploy (Step 3) | +| `scripts/start-apiserver.sh` not generated | No binary at `/tmp/apiserver` (`deploy-ha.sh` logs a `Warning:`, not an error) — this is a separate Go build, not part of `$BINARY`'s meson build dir | `cd src/frontend && make build-server`, then redeploy (Step 3) | +| Apiserver exits immediately / `Failed to initialize configuration files` | `$TOP/apiserver/config/app.yaml` failed to parse or is missing a required field (e.g. `file_upload.max_size`) — most likely the shipped `src/frontend/apiserver/config/app.yaml` template this is copied from changed shape | Inspect `$TOP/logs/apiserver.log` / `$TOP/apiserver/logs/logger_error.log`; diff `$TOP/apiserver/config/app.yaml` against the shipped template | +| Apiserver boots but never logs `Backend connection registered` | `app.yaml`'s `backend.peers` doesn't match the controller cluster (stale redeploy), or no controller is up yet | Confirm `$TOP/apiserver/config/app.yaml`'s `backend.peers` matches `$TOP/scripts/start.sh`'s `--peers`; start (or wait for) a controller first | +| `start-all.sh` reports "no primary election observed... proceeding anyway" | Same causes as `election_timeout` above — no controller became primary within 30s | Check controller logs directly (Step 5); worker/scheduler/apiserver may still fail to connect until a primary exists | diff --git a/.kiro/specs/resource-value-verification/tasks.md b/.kiro/specs/resource-value-verification/tasks.md index ca91608c..de35228e 100644 --- a/.kiro/specs/resource-value-verification/tasks.md +++ b/.kiro/specs/resource-value-verification/tasks.md @@ -32,7 +32,7 @@ - _Depends: 2.1_ - [ ] 3. Core: apiserver local deployment capability -- [ ] 3.1 (P) Extend `xce-deploy-ha`'s `deploy-ha.sh` with a conditional apiserver step and a combined `start-all.sh` +- [x] 3.1 (P) Extend `xce-deploy-ha`'s `deploy-ha.sh` with a conditional apiserver step and a combined `start-all.sh` - In `deploy-ha.sh` Step 3, add a new conditional branch in the same *opt-in-by-presence* spirit as the existing worker/scheduler branch: check for a pre-built apiserver binary at its own Go-build output path (e.g. `/tmp/apiserver`, `make build-server`'s default) — NOT alongside `$BINARY`, since apiserver is a separate build system, not the same meson build dir worker/scheduler share; `deploy-ha.sh` does not invoke `make build-server` itself, building it remains a caller prerequisite (document in SKILL.md's Prerequisites section, same pattern as the existing `/xce-build controller` note). If present, generate `$TOP/scripts/start-apiserver.sh` plus a minimal `logger.yaml` (modeled on `make test-integration`'s `XCE_BIN`/`SRV_URL` convention) — functional auth/RBAC is NOT required (verification is trace-log-based, task 6.4; no REST query is ever issued against this instance), but `logger.yaml` MUST set `logging.level: debug` or task 4.2's trace line is silently never written; absent the binary, generate nothing, so existing callers are unaffected - Also in Step 3, always generate `$TOP/scripts/start-all.sh`: runs `start.sh` for the controller node(s), waits on Step 5's existing health-check, then runs whichever of `start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually generated for this deploy (skipping any that weren't) — start the scheduler with `-vvv` specifically (verbosity 3, matching `tlog_debug3`/`TRACE`), overriding SKILL.md's existing `-v`-only example for this feature's needs, or task 4.1's trace line is silently never written - Update `.claude/skills/xce-deploy-ha/SKILL.md` with a new "Step 4d" documenting the apiserver step (following the existing Step 4b pattern) and documenting `start-all.sh` as a combined shortcut for Steps 4/4b/4c @@ -118,5 +118,6 @@ ## Implementation Notes - Task 1.1: `resval_typeresolve.c`'s new function is `int resval_resolve_type(const char* name, int type_override)`, returning a `Value_u_which` tag or `RESVAL_TYPE_UNKNOWN` (-1, doesn't collide with the real 0-6 enum range). Task 2.1 should build on this exact signature rather than inventing a new one. `test.c`'s `set_resval()`/`cmd_resupdate()` were deliberately left untouched by task 1.1 — task 2.1 is the one that makes `set_resval()` a thin wrapper. - `controller_unit_test` (the big legacy Unity binary) has a pre-existing, unrelated flaky failure in `test_config_repl.c` (bug-1561, overlayfs `fdatasync errno=5` sandbox artifact) that shows up on every `/xce-build controller` run in this environment. It is NOT caused by this feature's changes — do not chase it in later tasks; cross-check `.wolf/buglog.json`'s bug-1561 entry before treating a `controller_unit_test` failure as real, and confirm via `git status` that `test_config_repl.c`/`config_repl.c` are untouched by the current task's diff. +- Task 3.1: `src/frontend/Makefile`'s `build-server` target does not work standalone in this environment — it never sets `CGO_CFLAGS`/`CGO_LDFLAGS` the way `test-backend` does, failing with `fatal error: client.h: No such file or directory`. Pre-existing, out-of-boundary for this feature (confirmed unrelated to any of this feature's changes). To build a real apiserver binary for a live smoke test, export the same CGO flags `make test-backend` derives before running `go build`. Worth a separate bug-log entry / follow-up fix to the Makefile itself, not fixed here. Also found and worked around (config-only, no source change): `main.go`'s `RegisterToBackend` gate checks `BackendYamlConfig.Address` before consulting `Peers`, so a generated config needs `address`/`port` populated from the first `--peers` candidate, not just `peers` itself (bug-1571). - Task 2.2: `src/controller/meson.build` compiles `test.c` only into the `xce_test` binary (`xce_test_src`), never into `controller_unit_test` (`controller_unit_test_src`) — a `test.c`-only diff structurally cannot regress `controller_unit_test`'s results. Also: `controller_unit_test` is built with ASAN `detect_leaks=0` as pre-existing, repo-wide meson policy (not something any task in this feature turned off) — to actually check a `test.c` change for new leaks, run `xce_test` directly with leak detection left ON, don't rely on the meson suite's leak-detection status. - Task 2.1 (round 1 REJECTED, round 2 fixed): refactoring a getopt-loop's flush/finalization timing (needed to let a trailing flag like `-t` apply to its preceding `-n`/`-v` pair) can silently drop an unrelated existing guard (here: duplicate-`-v` rejection) that depended on the old immediate-finalization timing. When touching `cmd_resupdate()`-shaped CLI parsers in this codebase again, explicitly re-trace EVERY branch of the old getopt loop against the new one, not just the branches directly related to the new flag — a "traced every path" claim is only as good as the list of paths actually enumerated. `xce_test` also needs `ASAN_OPTIONS=detect_leaks=0` for reliable non-interactive stdin-piped smoke testing (its default leak-detection exit path can swallow buffered stdout). diff --git a/src/tools/deploy-ha.sh b/src/tools/deploy-ha.sh index cea834c5..004d82bb 100755 --- a/src/tools/deploy-ha.sh +++ b/src/tools/deploy-ha.sh @@ -96,6 +96,27 @@ if [ ! -x "$SCHED_BIN" ]; then SCHED_BIN="" fi +# Apiserver binary check -- same opt-in-by-presence spirit as WORKER_BIN/ +# SCHED_BIN above, but NOT a same-build-generation sibling lookup: the +# apiserver is a SEPARATE build system (Go, `make build-server` in +# src/frontend/Makefile), not part of the meson build dir $BINARY lives in, +# so there is no "$BINARY_BUILD_DIR/apiserver/..." to check. Its own build +# output always lands at a fixed path (`/tmp/apiserver`, build-server's +# `go build -o` target) regardless of which meson profile/build dir $BINARY +# came from. deploy-ha.sh does NOT invoke `make build-server` itself -- +# building it remains a caller prerequisite (same "run /xce-build controller +# first if stale" convention SKILL.md's Prerequisites section already +# documents for the controller, just a different build command). Soft-check +# only, same reasoning as worker/scheduler: xce-test-ha never builds an +# apiserver and calls this same script, so a missing binary must not fail +# the deploy -- it just skips generating scripts/start-apiserver.sh, with a +# warning. +APISERVER_BIN="/tmp/apiserver" +if [ ! -x "$APISERVER_BIN" ]; then + echo "Warning: no apiserver binary found at $APISERVER_BIN -- skipping scripts/start-apiserver.sh generation (run 'cd src/frontend && make build-server' first if you need it)" + APISERVER_BIN="" +fi + # xceadmin wheel (same-build-generation sibling again): /xce-build xceadmin # leaves one at /xceadmin/dist/xceadmin-*.whl (meson-build.sh). # Installed below into its OWN venv under $TOP, not just pointed at @@ -241,6 +262,119 @@ EOF chmod +x "$TOP/scripts/start-scheduler.sh" fi +# Generate apiserver config + start script -- only if a pre-built apiserver +# binary was found above. This is NOT an exact mirror of the worker/ +# scheduler conditional: those two just launch an existing meson-build +# binary with --peers; the apiserver additionally needs its own topDir +# (`-d`) carrying config/{app.yaml,auth.yaml,rbac.yaml,logger.yaml} before +# it will even boot (utils.PromptForTopDir/InitConf/InitLogger all read +# from "$TOPDIR/config/*.yaml"), so this block also generates that config, +# not just a launcher script. +# +# The apiserver's own topDir is $TOP/apiserver (its own config/ subdir, +# separate from the controller's $TOP/controller/ctrl-N/conf -- they must +# not collide). Its log file, however, is pointed via logger.yaml's +# outputPath at an ABSOLUTE path directly under $TOP/logs -- logger.go only +# prepends "/logs/" to a RELATIVE outputPath, so an absolute one +# here lands the apiserver's log in the SAME shared logs/ directory +# ctrl-N.log/scheduler.log already use, matching this deploy's existing +# logs/ convention (this is also where a later verification skill greps +# for the apiserver's resource-update trace line). +if [ -n "$APISERVER_BIN" ]; then + APISERVER_SRC_CONFIG="$SCRIPT_DIR/../frontend/apiserver/config" + mkdir -p "$TOP/apiserver/config" + + # auth.yaml / rbac.yaml: copied verbatim from the repo's own shipped + # dev-mode templates (config_file provider, built-in admin/basic_user + # roles) -- functional auth/RBAC is NOT required for this deploy (no + # REST query is ever issued against this apiserver instance), only + # enough for auth.InitAuthProvider()/controllers.LoadRBACConfig() to + # load successfully at boot without erroring. + cp "$APISERVER_SRC_CONFIG/auth.yaml" "$TOP/apiserver/config/auth.yaml" + cp "$APISERVER_SRC_CONFIG/rbac.yaml" "$TOP/apiserver/config/rbac.yaml" + + # app.yaml: copied verbatim from the shipped template (keeps every + # required field -- e.g. file_upload.max_size, which InitConf() + # strictly validates and errors out on if missing/malformed -- in sync + # with whatever the repo's own default currently is), then its + # trailing `backend:` section (the last top-level section in the + # shipped file today) is stripped and replaced with one pointing at + # THIS deploy's own controller cluster: the identical + # ctrl-N=host:port candidate list every other generated script here + # already uses, in the exact "[name=]host:port" syntax + # RegisterToBackend's xce_parse_peers expects (backend.peers takes + # priority over the legacy single address/port fallback -- see + # apiserver/backend/config.go). Without this, BackendYamlConfig.Address + # stays empty and main.go never calls RegisterToBackend at all -- + # the apiserver would boot in permanent stub mode and + # OnResourceLocationUpdated would never fire. + sed '/^backend:/,$d' "$APISERVER_SRC_CONFIG/app.yaml" > "$TOP/apiserver/config/app.yaml" + APISERVER_PEERS_YAML="" + for i in "${!PEER_ARRAY[@]}"; do + n=$((i+1)) + APISERVER_PEERS_YAML+=" - \"ctrl-$n=${PEER_ARRAY[$i]}\" +" + done + # address/port are still populated (first --peers candidate), even + # though `peers` (non-empty) is what RegisterToBackend actually uses to + # pick a connection candidate -- confirmed live: main.go's own gate for + # whether to call RegisterToBackend AT ALL is `BackendYamlConfig.Address + # != ""`, checked BEFORE Peers is ever consulted (cmd/main.go:58). A + # blank address here (even with a fully-populated non-empty peers list) + # left the apiserver silently booting in permanent stub mode ("Backend + # not configured") with RegisterToBackend never called -- caught during + # this task's own live verification, not a hypothetical. + APISERVER_FIRST_HOST="${PEER_ARRAY[0]%%:*}" + APISERVER_FIRST_PORT="${PEER_ARRAY[0]##*:}" + cat >> "$TOP/apiserver/config/app.yaml" << EOF +backend: + address: "$APISERVER_FIRST_HOST" + port: $APISERVER_FIRST_PORT + from: "xce_apiserver" + key: "keepsecret\$" + peers: +$APISERVER_PEERS_YAML +EOF + + # logger.yaml: NOT copied from the shipped template -- generated fresh + # so `logging.level: debug` is explicit and load-bearing here, rather + # than incidentally inherited from whatever the shipped template's + # default happens to be today. A later trace-log addition at the + # apiserver's resource-update storage point is only ever emitted via + # logger.Debug(...) (zap); at any less-verbose configured level it is + # silently dropped, which would look identical to a genuine + # propagation failure to anything polling this log. + cat > "$TOP/apiserver/config/logger.yaml" << EOF +logging: + level: "debug" + encoding: console + outputPath: "$TOP/logs/apiserver.log" + file: + maxSize: 100 + maxBackups: 10 + maxAge: 14 + compress: true + encoderConfig: + messageKey: "message" + levelKey: "level" + timeKey: "timestamp" + nameKey: "logger" + callerKey: "caller" + functionKey: "func" + stacktraceKey: "stacktrace" + timeEncoder: "ISO8601" + levelEncoder: "capital" +EOF + + cat > "$TOP/scripts/start-apiserver.sh" << EOF +#!$GEN_BASH +set -e +mkdir -p $TOP/logs +$APISERVER_BIN -d $TOP/apiserver \$* +EOF + chmod +x "$TOP/scripts/start-apiserver.sh" +fi + # Install xceadmin into THIS deployment ($TOP/xceadmin-site), not just # pointed at $XCEADMIN's build-dir .venv resolved above -- a real, standalone # copy, so the deployment keeps a working xceadmin long after the build dir @@ -297,8 +431,84 @@ EOF chmod +x "$TOP/scripts/xceadmin" fi +# Generate start-all.sh -- ALWAYS (unlike start-worker.sh/start-scheduler.sh/ +# start-apiserver.sh above, this one is not conditional on any optional +# binary's presence). It automates the exact manual recipe SKILL.md's Steps +# 4/4b/4c already document a human running by hand: start every controller +# node, wait for the same primary-election health-check Step 5 documents, +# then start whichever of start-worker.sh/start-scheduler.sh/ +# start-apiserver.sh were actually generated for THIS deploy -- checked at +# the generated script's OWN runtime via [ -x ... ], not baked in as a +# deploy-time boolean, so start-all.sh is a single self-contained artifact +# that correctly no-ops a step whose binary wasn't available, with no +# regeneration needed if e.g. a stale copy of start-all.sh is run after +# manually deleting one of the optional scripts. +# +# Every component is started via `setsid nohup ... /dev/null +# 2>&1 & disown`, not a plain `&` -- reparents each process to init so it +# survives start-all.sh's own exit (needed since this script is meant to be +# run once and return, leaving the cluster running for later interactive +# use, exactly like this skill's own SKILL.md already recommends for +# Agent-subagent dispatch of Steps 1-5 and xce-test-ha's soak driver). +# +# The scheduler is started with -vvv specifically (not the -v this file's +# own start-scheduler.sh body already bakes in unconditionally) so its +# resulting verbosity is >= 3 (`debug3`) -- verbose_to_loglevel() caps at +# "debug3" for any count above 3, so stacking with the wrapper's own -v is +# harmless, but passing anything less than vvv here risks the trace log a +# later resource-verification feature depends on being silently never +# written at the daemon's default verbosity. +cat > "$TOP/scripts/start-all.sh" << EOF +#!$GEN_BASH +set -e + +echo "=== start-all.sh: starting controller node(s) ===" +for i in \$(seq 1 $N); do + setsid nohup "$GEN_BASH" "$TOP/scripts/start.sh" ctrl-\$i -vvv /dev/null 2>&1 & + disown + sleep 0.5 +done + +echo "=== start-all.sh: waiting for controller health-check (up to 30s) ===" +HEALTHY=0 +for i in \$(seq 1 30); do + if grep -lE "CTRL_PRIMARY" $TOP/logs/ctrl-*.log >/dev/null 2>&1; then + HEALTHY=1 + break + fi + sleep 1 +done +if [ "\$HEALTHY" -eq 0 ]; then + echo "Warning: no primary election observed in $TOP/logs/ctrl-*.log within 30s -- proceeding anyway (worker/scheduler/apiserver may fail to connect)" +else + echo "Controller cluster healthy (primary elected)." +fi + +if [ -x "$TOP/scripts/start-worker.sh" ]; then + echo "=== start-all.sh: starting worker ===" + setsid nohup "$GEN_BASH" "$TOP/scripts/start-worker.sh" worker-1 -vvv /dev/null 2>&1 & + disown +fi + +if [ -x "$TOP/scripts/start-scheduler.sh" ]; then + echo "=== start-all.sh: starting scheduler (-vvv) ===" + setsid nohup "$GEN_BASH" "$TOP/scripts/start-scheduler.sh" -vvv /dev/null 2>&1 & + disown +fi + +if [ -x "$TOP/scripts/start-apiserver.sh" ]; then + echo "=== start-all.sh: starting apiserver ===" + setsid nohup "$GEN_BASH" "$TOP/scripts/start-apiserver.sh" /dev/null 2>&1 & + disown +fi + +echo "=== start-all.sh: done -- cluster left running ===" +EOF +chmod +x "$TOP/scripts/start-all.sh" + echo "Deployed $N-node cluster to $TOP (tag: $TAG)" if [ -n "$XCEADMIN_WHEEL" ]; then echo "xceadmin installed at $TOP/xceadmin-site -- use it via $TOP/scripts/xceadmin, e.g.:" echo " $TOP/scripts/xceadmin config push $TOP/controller/ctrl-1/conf/cluster.bin.$TAG --controller ${PEER_ARRAY[0]}" fi +echo "Run \"$GEN_BASH\" $TOP/scripts/start-all.sh to start controller(s) plus whichever of worker/scheduler/apiserver were generated for this deploy."