forked from lei.chen/myxce
Compare commits
2 commits
bbe8ed3335
...
c1e9265b14
| Author | SHA1 | Date | |
|---|---|---|---|
|
c1e9265b14 | ||
|
|
68ab5a071a |
6 changed files with 1429 additions and 0 deletions
|
|
@ -15037,5 +15037,27 @@
|
||||||
"related_bugs": [],
|
"related_bugs": [],
|
||||||
"occurrences": 1,
|
"occurrences": 1,
|
||||||
"last_seen": "2026-08-13"
|
"last_seen": "2026-08-13"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "bug-1514",
|
||||||
|
"timestamp": "2026-08-21",
|
||||||
|
"error_message": "换 branch 后全部作业再次 PEND 不调度：scheduler 持续 'unsatisfy_unmet resource 0 (slots) shortfall 1.000000'，resval -n slots -m test01 返回 0。与 bug-1494 同症状。",
|
||||||
|
"file": "src/configs/hosts.yaml + deploy/run/controller/ctrl-1/conf/cluster.bin.<tag>",
|
||||||
|
"root_cause": "操作时序：hosts.yaml 15:27:08 加了 test01 hostconfig（max_slot: 8，用于修复 bug-1494 的 slots=0），但 cluster.bin 是 15:24:25 构建的——不含 test01 段。controller 加载旧 cluster.bin 后 worker test01 join 落到 default hostconfig（max_slot: -1 → apply_join_state 回落 w->host.attrs->maxcpus，worker 上报 maxcpus 为垃圾/0）→ slots=0 → scheduler 总 slots=0 → 全部 PEND。换 branch 重建二进制后只做了 config deploy（旧 hosts.yaml），未注意 hosts.yaml 工作区未提交修改未进 cluster.bin。",
|
||||||
|
"fix": "重新 ./all.sh config deploy（用最新 hosts.yaml 重建 cluster.bin.d7c0dc804fc8）并重启 controller；之后 restart worker 让其重新 join（新 hostconfig max_slot:8 → slots=8）；restart apiserver（重启 controller 时 apiserver ASAN 'Joining already joined thread' 崩溃）。验证：resval slots@test01=8.0，新作业 RUN，之前 PEND 的 sleep 221/222 也被补调度 RUN。教训：改 src/configs/*.yaml 后必须重新 config deploy，cluster.bin 是编译产物不会自动跟随源。",
|
||||||
|
"tags": [
|
||||||
|
"pend",
|
||||||
|
"slots-zero",
|
||||||
|
"cluster-bin-stale",
|
||||||
|
"config-deploy",
|
||||||
|
"operational",
|
||||||
|
"bug-1494-symptom",
|
||||||
|
"fixed"
|
||||||
|
],
|
||||||
|
"related_bugs": [
|
||||||
|
"bug-1494"
|
||||||
|
],
|
||||||
|
"occurrences": 1,
|
||||||
|
"last_seen": "2026-08-21"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
@ -7399,3 +7399,9 @@
|
||||||
| 02:1x | Added human-readable jobid (job_id_human/human_id) to apiserver responses (backend.FormatJobID reuse); renamed SSE perm event:subscribe->event:view; updated docs (left_scheduler_work, design doc v1.3, cerebrum) | src/frontend/apiserver, src/frontend/schedulerclient, docs | verified live (submit + get_job return 1-0-... form); all tests pass; pushed | ~90k tok |
|
| 02:1x | Added human-readable jobid (job_id_human/human_id) to apiserver responses (backend.FormatJobID reuse); renamed SSE perm event:subscribe->event:view; updated docs (left_scheduler_work, design doc v1.3, cerebrum) | src/frontend/apiserver, src/frontend/schedulerclient, docs | verified live (submit + get_job return 1-0-... form); all tests pass; pushed | ~90k tok |
|
||||||
| 06:21 | bug-1512 多 worker 过派发修复：handler.c code<0 才 re-PEND；24 作业 8/8/8 全 DONE；单测+提交 e4aaef4 | handler.c test_handlers.c | fixed+verified | ~2k |
|
| 06:21 | bug-1512 多 worker 过派发修复：handler.c code<0 才 re-PEND；24 作业 8/8/8 全 DONE；单测+提交 e4aaef4 | handler.c test_handlers.c | fixed+verified | ~2k |
|
||||||
| 09:1x | PEND reason 全链路补全（bug-1513）：controller SETPREASON hook 推送 + apiserver 转换层填充 + JOB_DEFAULT_COLUMN；集群验证 resreq type==nosuchhost → host_reasons [{t1,3},{t2,3},{t3,3}]；提交 ac6131d | apiserver.c convert.go convert_pure.go convert_test.go models/jobs.go | verified | ~40k |
|
| 09:1x | PEND reason 全链路补全（bug-1513）：controller SETPREASON hook 推送 + apiserver 转换层填充 + JOB_DEFAULT_COLUMN；集群验证 resreq type==nosuchhost → host_reasons [{t1,3},{t2,3},{t3,3}]；提交 ac6131d | apiserver.c convert.go convert_pure.go convert_test.go models/jobs.go | verified | ~40k |
|
||||||
|
|
||||||
|
## Session: 2026-08-21 15:3x
|
||||||
|
|
||||||
|
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||||
|
|------|--------|---------|---------|--------|
|
||||||
|
| 15:3x | 修复换 branch 后全部作业 PEND（bug-1514，bug-1494 症状复发）：根因=hosts.yaml 15:27 加的 test01 hostconfig(max_slot:8) 未重新 config deploy（cluster.bin 15:24 构建不含），worker join 落 default(max_slot:-1→回落 maxcpus 垃圾值)→slots=0。重新 config deploy(cluster.bin.d7c0dc804fc8)+restart worker+restart apiserver（controller 重启时 apiserver ASan 崩溃）→ slots@test01=8.0，PEND 作业全部补调度 RUN。教训：改 src/configs/*.yaml 后必须重新 config deploy，cluster.bin 是编译产物不自动跟随源 | src/configs/hosts.yaml, all.sh, .wolf/buglog.json | fixed+verified | ~40k |
|
||||||
|
|
|
||||||
795
all.sh
Executable file
795
all.sh
Executable file
|
|
@ -0,0 +1,795 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# prepare env begin
|
||||||
|
# sudo chmod 775 /nix/var/nix/daemon-socket
|
||||||
|
# sudo systemctl restart nix-daemon
|
||||||
|
# prepare env end
|
||||||
|
|
||||||
|
# customzie begin
|
||||||
|
#g_code_base="/home/jsxing/allwork/git/xce"
|
||||||
|
g_code_base="/home/jsxing/allwork/git/myxce"
|
||||||
|
|
||||||
|
# services registry: name|binary path, managed by start/stop/restart/status
|
||||||
|
g_services=(
|
||||||
|
"controller|${g_code_base}/src/build/controller/xce_controller"
|
||||||
|
"scheduler|${g_code_base}/src/build/scheduler/xce_scheduler"
|
||||||
|
"worker|${g_code_base}/src/build/worker/xcecd"
|
||||||
|
"apiserver|${g_code_base}/src/build/frontend/apiserver/xce_apiserver"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Unified deployment root: ALL configs, logs and runtime state live under one
|
||||||
|
# directory next to all.sh, so nothing references system dirs (/opt/xce) or
|
||||||
|
# stale repo paths. Layout:
|
||||||
|
# deploy/conf/ worker.yaml + other generated configs
|
||||||
|
# deploy/run/ pid files, certs, controller workdir, apiserver topdir
|
||||||
|
# deploy/logs/ controller/scheduler/worker log files
|
||||||
|
g_deploy_dir="${g_code_base}/deploy"
|
||||||
|
g_conf_dir="${g_deploy_dir}/conf" # generated config files (worker.yaml, ...)
|
||||||
|
g_run_dir="${g_deploy_dir}/run" # pid files, certs, controller/apiserver data
|
||||||
|
g_log_dir="${g_deploy_dir}/logs" # log files
|
||||||
|
|
||||||
|
# Log verbosity for ALL components. One of: info | debug | debug2 | debug3.
|
||||||
|
# controller/scheduler: mapped to -v / -vv / -vvv (argp --verbose)
|
||||||
|
# worker (xcecd): written into worker.yaml log_level field
|
||||||
|
# apiserver: written into logger.yaml level (zap; debug2/3 -> debug)
|
||||||
|
g_loglevel="debug3"
|
||||||
|
|
||||||
|
# Per-job worker binary install dir. xcecd_worker is exec'd by systemd AS the
|
||||||
|
# job's exec user (e.g. admin), so it must live somewhere every user can reach.
|
||||||
|
# The repo path under /home/<user> is usually 750 and blocks other users ->
|
||||||
|
# exec fails with "Permission denied" (sh exit 126) and every job dies instantly.
|
||||||
|
# /opt/xce/bin is the canonical XCE layout. svc_prepare_worker installs it there
|
||||||
|
# via sudo and svc_env_worker points XCE_SERVER_DIR at it. Can be overridden.
|
||||||
|
g_worker_bin_dir="/opt/xce/bin"
|
||||||
|
# customize end
|
||||||
|
|
||||||
|
# usage begin
|
||||||
|
|
||||||
|
# ./all.sh build -c # 清理
|
||||||
|
# ./all.sh build -d # 开发构建（带耗时统计）
|
||||||
|
# ./all.sh build -p # 生产构建（带耗时统计）
|
||||||
|
# ./all.sh config deploy # 构建+部署集群配置（cluster.bin）并重启 controller
|
||||||
|
# ./all.sh config backup # 备份所有服务配置到 ~/all_configs
|
||||||
|
# ./all.sh start [svc] # 启动服务（默认全部）：controller scheduler worker apiserver
|
||||||
|
# ./all.sh stop [svc] # 停止服务（默认全部）
|
||||||
|
# ./all.sh restart [svc] # 重启服务（默认全部）
|
||||||
|
# ./all.sh status [svc] # 查看服务状态（默认全部）
|
||||||
|
|
||||||
|
# -d/-p：增量编译，只有 xceadmin 每次固定重跑（~11s）+ 首次/源码变更时全量。初次全量约 60s。
|
||||||
|
# -c：彻底，重建一次即可得到干净状态。
|
||||||
|
# svc 取值为：controller / scheduler / worker，可传多个，如 ./all.sh start controller scheduler
|
||||||
|
# worker (xcecd) 需要 root，通过 sudo 启动/停止；sudo 需免密或已有缓存凭证（sudo -n）
|
||||||
|
# 无 cluster.bin 时 controller 会拒绝 worker join（EINVAL）——用 ./all.sh config deploy 部署
|
||||||
|
# 所有配置/logs/run 统一在 ${g_deploy_dir}（deploy/）下：
|
||||||
|
# worker.yaml 生成在 ${g_conf_dir}/worker.yaml（不再用 /opt/xce/conf/worker.yaml）
|
||||||
|
# controller 配置在 ${g_run_dir}/controller/ctrl-1/conf
|
||||||
|
# apiserver 配置在 ${g_run_dir}/apiserver/config
|
||||||
|
# 日志在 ${g_log_dir}/，pid 在 ${g_run_dir}/
|
||||||
|
|
||||||
|
# usage end
|
||||||
|
|
||||||
|
# log in a function, please print the calling function name
|
||||||
|
function _in() {
|
||||||
|
echo "[$(date +'%Y-%m-%d %H:%M:%S')] [${FUNCNAME[1]}] $1"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Map g_loglevel -> argp --verbose flag count (-v/-vv/-vvv) for controller and
|
||||||
|
# scheduler (they call verbose_to_loglevel(); see src/lib/utils.c).
|
||||||
|
function _verbose_flags() {
|
||||||
|
case "${g_loglevel:-info}" in
|
||||||
|
debug3) echo "-vvv" ;;
|
||||||
|
debug2) echo "-vv" ;;
|
||||||
|
debug) echo "-v" ;;
|
||||||
|
*) echo "" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Map g_loglevel -> worker.yaml log_level value (xcecd level_to_int()).
|
||||||
|
function _worker_loglevel() {
|
||||||
|
case "${g_loglevel:-info}" in
|
||||||
|
debug|debug2|debug3|notice|warn|error|fatal|off) echo "${g_loglevel}" ;;
|
||||||
|
*) echo "info" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Map g_loglevel -> apiserver logger.yaml level (zap; only debug/info/warn/...).
|
||||||
|
function _apiserver_loglevel() {
|
||||||
|
case "${g_loglevel:-info}" in
|
||||||
|
debug|debug2|debug3) echo "debug" ;;
|
||||||
|
*) echo "info" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# run a command and log the command and its output
|
||||||
|
function run_cmd() {
|
||||||
|
local datestr=$(date "+%Y-%m-%d %H:%M:%S")
|
||||||
|
local cmd="$1 2>&1"
|
||||||
|
echo "[${datestr}]executing: ${1}"
|
||||||
|
|
||||||
|
echo "${cmd}" | awk '
|
||||||
|
function cmd( c )
|
||||||
|
{
|
||||||
|
while( (c|getline foo) > 0 )
|
||||||
|
printf( "%s\n", foo )
|
||||||
|
close( c );
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
cmd( $0 );
|
||||||
|
}
|
||||||
|
'
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---- service management begin ----
|
||||||
|
|
||||||
|
# Return the startup arguments for a service.
|
||||||
|
# Override these in the customize section to fit your deployment.
|
||||||
|
function svc_args_controller() {
|
||||||
|
echo "--myname ctrl-1 --peers ctrl-1=localhost:9990 \
|
||||||
|
--workdir ${g_run_dir}/controller --certdir ${g_run_dir}/certs \
|
||||||
|
--logfile ${g_log_dir}/controller.log --vcpus 4 $(_verbose_flags)"
|
||||||
|
}
|
||||||
|
function svc_args_scheduler() {
|
||||||
|
echo "-l ${g_log_dir}/scheduler.log $(_verbose_flags)"
|
||||||
|
}
|
||||||
|
function svc_args_worker() {
|
||||||
|
# NOTE: when -c is given xcecd IGNORES -l/-p/-m (config file wins), so the
|
||||||
|
# log dir is fully controlled by ${g_conf_dir}/worker.yaml (log_dir field).
|
||||||
|
echo "-c ${g_conf_dir}/worker.yaml -n localhost01"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Extra environment for the worker (xcecd) process. XCE_SERVER_DIR tells xcecd
|
||||||
|
# where to find xcecd_worker for per-job workers; it must point at a directory
|
||||||
|
# every job exec user can reach (see g_worker_bin_dir). Without it xcecd infers
|
||||||
|
# the dir from its own binary path (under /home/<user>) which job users can't
|
||||||
|
# traverse -> exec fails "Permission denied" / exit 126. svc_start prepends
|
||||||
|
# these via `env` for sudo-launched services.
|
||||||
|
function svc_env_worker() {
|
||||||
|
echo "XCE_SERVER_DIR=${g_worker_bin_dir}"
|
||||||
|
}
|
||||||
|
function svc_args_apiserver() {
|
||||||
|
echo "-d ${g_run_dir}/apiserver"
|
||||||
|
}
|
||||||
|
|
||||||
|
# services that must run as root (launched/stopped via sudo)
|
||||||
|
function svc_needs_sudo() {
|
||||||
|
case "$1" in
|
||||||
|
worker) return 0 ;;
|
||||||
|
*) return 1 ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Optional per-service prepare hook, called by svc_start before launch.
|
||||||
|
# controller: ensure the self-signed TLS cert/key exist (like deploy-ha.sh).
|
||||||
|
function svc_prepare_controller() {
|
||||||
|
local certdir="${g_run_dir}/certs"
|
||||||
|
mkdir -p "${certdir}"
|
||||||
|
if [ ! -f "${certdir}/xce.crt" ] || [ ! -f "${certdir}/xce.key" ]; then
|
||||||
|
_in "Generating self-signed TLS cert/key in ${certdir}"
|
||||||
|
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
|
||||||
|
-keyout "${certdir}/xce.key" -out "${certdir}/xce.crt" \
|
||||||
|
-subj "/CN=xce-dev-test" 2>/dev/null
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# apiserver: ensure <topdir>/config/app.yaml + logger.yaml exist
|
||||||
|
# (app.yaml schema-valid, backend -> controller on 9990). Without them the
|
||||||
|
# Go binary refuses to start.
|
||||||
|
function svc_prepare_apiserver() {
|
||||||
|
local topdir="${g_run_dir}/apiserver"
|
||||||
|
local confdir="${topdir}/config"
|
||||||
|
mkdir -p "${confdir}"
|
||||||
|
if [ ! -f "${confdir}/logger.yaml" ]; then
|
||||||
|
_in "Generating default ${confdir}/logger.yaml (level=$(_apiserver_loglevel))"
|
||||||
|
cat > "${confdir}/logger.yaml" <<EOF
|
||||||
|
logging:
|
||||||
|
level: "$(_apiserver_loglevel)"
|
||||||
|
encoding: "console"
|
||||||
|
outputPath: "apiserver.log"
|
||||||
|
file:
|
||||||
|
maxSize: 64
|
||||||
|
maxBackups: 10
|
||||||
|
maxAge: 30
|
||||||
|
compress: true
|
||||||
|
encoderConfig:
|
||||||
|
messageKey: "msg"
|
||||||
|
levelKey: "level"
|
||||||
|
timeKey: "ts"
|
||||||
|
nameKey: "logger"
|
||||||
|
callerKey: "caller"
|
||||||
|
functionKey: "func"
|
||||||
|
stacktraceKey: "stacktrace"
|
||||||
|
timeEncoder: "iso8601"
|
||||||
|
levelEncoder: "capital"
|
||||||
|
EOF
|
||||||
|
else
|
||||||
|
# keep existing logger.yaml but sync level to the configured g_loglevel
|
||||||
|
local want="$(_apiserver_loglevel)"
|
||||||
|
if ! grep -q '^ level: "'"${want}"'"' "${confdir}/logger.yaml"; then
|
||||||
|
_in "updating ${confdir}/logger.yaml level -> ${want}"
|
||||||
|
sed -i 's/^ level: ".*"/ level: "'"${want}"'"/' "${confdir}/logger.yaml"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
if [ ! -f "${confdir}/rbac.yaml" ]; then
|
||||||
|
if [ -f "${g_code_base}/src/frontend/apiserver/config/rbac.yaml" ]; then
|
||||||
|
_in "Copying sample rbac.yaml"
|
||||||
|
cp "${g_code_base}/src/frontend/apiserver/config/rbac.yaml" \
|
||||||
|
"${confdir}/rbac.yaml"
|
||||||
|
else
|
||||||
|
_in "Generating minimal rbac.yaml"
|
||||||
|
cat > "${confdir}/rbac.yaml" <<'EOF'
|
||||||
|
permissions: []
|
||||||
|
roles: {}
|
||||||
|
users: {}
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
if [ ! -f "${confdir}/auth.yaml" ]; then
|
||||||
|
if [ -f "${g_code_base}/src/frontend/apiserver/config/auth.yaml" ]; then
|
||||||
|
_in "Copying sample auth.yaml"
|
||||||
|
cp "${g_code_base}/src/frontend/apiserver/config/auth.yaml" \
|
||||||
|
"${confdir}/auth.yaml"
|
||||||
|
else
|
||||||
|
_in "Generating minimal auth.yaml"
|
||||||
|
cat > "${confdir}/auth.yaml" <<'EOF'
|
||||||
|
provider: config_file
|
||||||
|
config_file:
|
||||||
|
users:
|
||||||
|
- username: admin
|
||||||
|
password: admin
|
||||||
|
roles: [cluster_admin]
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
if [ ! -f "${confdir}/app.yaml" ]; then
|
||||||
|
_in "Generating default ${confdir}/app.yaml"
|
||||||
|
cat > "${confdir}/app.yaml" <<'EOF'
|
||||||
|
app:
|
||||||
|
name: "xce_apiserver"
|
||||||
|
runmode: "release"
|
||||||
|
http_port: 8080
|
||||||
|
version: "1.0.0"
|
||||||
|
|
||||||
|
security:
|
||||||
|
https:
|
||||||
|
enabled: false
|
||||||
|
port: 8443
|
||||||
|
cert_file: "config/ssl/server.crt"
|
||||||
|
key_file: "config/ssl/server.key"
|
||||||
|
|
||||||
|
authentication:
|
||||||
|
token:
|
||||||
|
type: "jwt"
|
||||||
|
secretKey: "xce-api-key"
|
||||||
|
token_expire_time: 86400
|
||||||
|
|
||||||
|
rate_limit:
|
||||||
|
global:
|
||||||
|
rate: 1000
|
||||||
|
burst: 2000
|
||||||
|
window: second
|
||||||
|
endpoints:
|
||||||
|
/xce/v1/auth/logon:
|
||||||
|
rate: 100
|
||||||
|
burst: 200
|
||||||
|
window: second
|
||||||
|
|
||||||
|
whitelist:
|
||||||
|
- "0.0.0.0/0"
|
||||||
|
- "::/0"
|
||||||
|
|
||||||
|
file_upload:
|
||||||
|
max_size: 100 MB
|
||||||
|
allowed_types:
|
||||||
|
- "application/x-tar"
|
||||||
|
- "application/zip"
|
||||||
|
|
||||||
|
job:
|
||||||
|
max_array_size: 1000
|
||||||
|
|
||||||
|
cache:
|
||||||
|
refresh_interval: 30
|
||||||
|
|
||||||
|
cors:
|
||||||
|
enabled: true
|
||||||
|
allowed_origins:
|
||||||
|
- "*"
|
||||||
|
|
||||||
|
backend:
|
||||||
|
address: "localhost"
|
||||||
|
port: 9990
|
||||||
|
from: "xce_apiserver"
|
||||||
|
key: "keepsecret$"
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# worker: generate worker.yaml under ${g_conf_dir} so the worker never reads a
|
||||||
|
# system path (/opt/xce/conf/worker.yaml) or a stale repo's log_dir. The
|
||||||
|
# log_dir field must point at our ${g_log_dir}; when -c is given xcecd ignores
|
||||||
|
# -l so this file is the single source of truth for the worker's log location.
|
||||||
|
function svc_prepare_worker() {
|
||||||
|
mkdir -p "${g_conf_dir}"
|
||||||
|
local wf="${g_conf_dir}/worker.yaml"
|
||||||
|
local lvl="$(_worker_loglevel)"
|
||||||
|
if [ -f "$wf" ]; then
|
||||||
|
# keep existing file but sync log_level to the configured g_loglevel
|
||||||
|
if ! grep -q "log_level: \"${lvl}\"" "$wf"; then
|
||||||
|
_in "updating worker.yaml log_level -> ${lvl}"
|
||||||
|
sed -i "s/log_level:.*/log_level: \"${lvl}\"/" "$wf"
|
||||||
|
fi
|
||||||
|
_in "worker config: ${wf} (log_level=${lvl})"
|
||||||
|
else
|
||||||
|
_in "Generating ${wf}"
|
||||||
|
cat > "$wf" <<EOF
|
||||||
|
---
|
||||||
|
workerconf:
|
||||||
|
management_list: ["localhost"]
|
||||||
|
controller_port: 9990
|
||||||
|
log_level: "${lvl}"
|
||||||
|
log_dir: "${g_log_dir}"
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Install xcecd_worker somewhere every job exec user can run it. The job's
|
||||||
|
# worker is launched by systemd with User=<exec user>; the repo path under
|
||||||
|
# /home/<user> is usually 750, so other users get "Permission denied"
|
||||||
|
# (exit 126) and jobs die instantly. xcecd finds the binary via
|
||||||
|
# XCE_SERVER_DIR (svc_env_worker), so it must match this install dir.
|
||||||
|
local worker_bin="${g_code_base}/src/build/worker/xcecd_worker"
|
||||||
|
local worker_dst="${g_worker_bin_dir}/xcecd_worker"
|
||||||
|
if [ -x "${worker_bin}" ]; then
|
||||||
|
if [ ! -f "${worker_dst}" ] || [ "${worker_bin}" -nt "${worker_dst}" ]; then
|
||||||
|
_in "installing ${worker_bin} -> ${worker_dst}"
|
||||||
|
sudo -n mkdir -p "${g_worker_bin_dir}"
|
||||||
|
sudo -n install -m 0755 "${worker_bin}" "${worker_dst}"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
_in "WARN: ${worker_bin} not found; jobs will fail to exec xcecd_worker" >&2
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Optional per-service precheck hook, called by svc_start BEFORE launching.
|
||||||
|
# controller: refuse to start without a deployed cluster.bin — without one the
|
||||||
|
# controller never becomes primary and apiserver's backend init fails with
|
||||||
|
# -10013 (bug-1514). Prints the active config when present.
|
||||||
|
function svc_check_controller() {
|
||||||
|
local confdir="${g_run_dir}/controller/ctrl-1/conf"
|
||||||
|
local cur ref bin
|
||||||
|
if [ ! -f "${confdir}/current" ]; then
|
||||||
|
_in "controller 未部署集群配置（缺少 ${confdir}/current）" >&2
|
||||||
|
_in "请先运行: ./all.sh config deploy # 从 src/configs 生成 cluster.bin 并部署" >&2
|
||||||
|
_in "然后再: ./all.sh start" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
ref="$(cat "${confdir}/current" 2>/dev/null)"
|
||||||
|
bin="${confdir}/${ref}"
|
||||||
|
if [ -n "$ref" ] && [ -f "$bin" ]; then
|
||||||
|
_in "controller 使用集群配置: ${bin} (current -> ${ref})"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
_in "controller 配置指针 current -> ${ref} 指向的文件不存在（${bin}）" >&2
|
||||||
|
_in "请重新运行: ./all.sh config deploy" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# list registered service names
|
||||||
|
function _svc_names() {
|
||||||
|
local entry n
|
||||||
|
for entry in "${g_services[@]}"; do
|
||||||
|
n="${entry%%|*}"
|
||||||
|
echo "$n"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# resolve binary path for a service name (empty if unknown)
|
||||||
|
function _svc_bin() {
|
||||||
|
local name="$1" entry n b
|
||||||
|
for entry in "${g_services[@]}"; do
|
||||||
|
n="${entry%%|*}"; b="${entry#*|}"
|
||||||
|
[ "$n" = "$name" ] && { echo "$b"; return 0; }
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
function _svc_pidfile() { echo "${g_run_dir}/$1.pid"; }
|
||||||
|
|
||||||
|
# check liveness of a service pid; uses sudo for root (sudo) services
|
||||||
|
function _svc_alive() {
|
||||||
|
local name="$1" pid="$2"
|
||||||
|
if svc_needs_sudo "$name"; then
|
||||||
|
sudo -n kill -0 "$pid" 2>/dev/null
|
||||||
|
else
|
||||||
|
kill -0 "$pid" 2>/dev/null
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# send a signal to a service pid; uses sudo for root (sudo) services
|
||||||
|
function _svc_kill() {
|
||||||
|
local name="$1" sig="$2" pid="$3"
|
||||||
|
if svc_needs_sudo "$name"; then
|
||||||
|
if [ -n "$sig" ]; then
|
||||||
|
sudo -n kill -"$sig" "$pid" 2>/dev/null
|
||||||
|
else
|
||||||
|
sudo -n kill "$pid" 2>/dev/null
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
if [ -n "$sig" ]; then
|
||||||
|
kill -"$sig" "$pid" 2>/dev/null
|
||||||
|
else
|
||||||
|
kill "$pid" 2>/dev/null
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# print the live pid of a service (or empty); cleanup stale pidfile
|
||||||
|
function _svc_pid() {
|
||||||
|
local name="$1" pidfile pid
|
||||||
|
pidfile="$(_svc_pidfile "$name")"
|
||||||
|
if [ -f "$pidfile" ]; then
|
||||||
|
pid="$(cat "$pidfile" 2>/dev/null)"
|
||||||
|
if [ -n "$pid" ] && _svc_alive "$name" "$pid"; then
|
||||||
|
echo "$pid"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
rm -f "$pidfile"
|
||||||
|
fi
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# resolve target services from args; default to all registered
|
||||||
|
function _svc_resolve() {
|
||||||
|
if [ $# -eq 0 ]; then
|
||||||
|
_svc_names
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
local name bin
|
||||||
|
for name in "$@"; do
|
||||||
|
bin="$(_svc_bin "$name")" || {
|
||||||
|
_in "unknown service: $name (valid: $(_svc_names | tr '\n' ' '))" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
echo "$name"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# start a single service (background, pidfile + log)
|
||||||
|
function svc_start() {
|
||||||
|
local name="$1" bin args pidfile log
|
||||||
|
bin="$(_svc_bin "$name")" || return 1
|
||||||
|
args="$(svc_args_${name} 2>/dev/null)" || args=""
|
||||||
|
|
||||||
|
[ -x "$bin" ] || { _in "binary not found: $bin (run ./all.sh build first)"; return 1; }
|
||||||
|
if [ -n "$(_svc_pid "$name")" ]; then
|
||||||
|
# already running: still report the active config (best-effort).
|
||||||
|
# stdout carries the "using config" line; stderr hints are muted here.
|
||||||
|
if declare -F "svc_check_${name}" >/dev/null 2>&1; then
|
||||||
|
"svc_check_${name}" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
_in "$name already running (pid $(_svc_pid "$name"))"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "${g_run_dir}" "${g_log_dir}"
|
||||||
|
pidfile="$(_svc_pidfile "$name")"
|
||||||
|
log="${g_log_dir}/${name}.log"
|
||||||
|
|
||||||
|
# optional per-service prepare hook (e.g. controller TLS certs)
|
||||||
|
if declare -F "svc_prepare_${name}" >/dev/null 2>&1; then
|
||||||
|
"svc_prepare_${name}" || return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# optional per-service precheck hook (e.g. controller requires cluster.bin)
|
||||||
|
if declare -F "svc_check_${name}" >/dev/null 2>&1; then
|
||||||
|
"svc_check_${name}" || return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if svc_needs_sudo "$name"; then
|
||||||
|
_in "Starting ${name} as root via sudo: ${bin} ${args}"
|
||||||
|
# optional per-service env (e.g. worker: XCE_SERVER_DIR so job workers
|
||||||
|
# can exec xcecd_worker from a world-accessible dir)
|
||||||
|
local envp=""
|
||||||
|
if declare -F "svc_env_${name}" >/dev/null 2>&1; then
|
||||||
|
envp="$(svc_env_${name})"
|
||||||
|
fi
|
||||||
|
# inner bash writes its own pid, then exec's into the binary so the
|
||||||
|
# pidfile holds the real (root) process pid; nohup guards SIGHUP.
|
||||||
|
# sudo -n = non-interactive; requires NOPASSWD or a cached credential.
|
||||||
|
sudo -n env ${envp} bash -c "echo \$\$ > '${pidfile}'; exec nohup '${bin}' ${args}" \
|
||||||
|
>>"${log}" 2>&1 < /dev/null &
|
||||||
|
else
|
||||||
|
_in "Starting ${name}: ${bin} ${args}"
|
||||||
|
setsid nohup ${bin} ${args} >>"${log}" 2>&1 < /dev/null &
|
||||||
|
echo $! > "${pidfile}"
|
||||||
|
fi
|
||||||
|
disown 2>/dev/null || true
|
||||||
|
|
||||||
|
sleep 1
|
||||||
|
local pid
|
||||||
|
pid="$(_svc_pid "$name")"
|
||||||
|
if [ -n "$pid" ]; then
|
||||||
|
_in "${name} started (pid $pid, log ${log})"
|
||||||
|
else
|
||||||
|
_in "${name} FAILED to start -- check ${log}"
|
||||||
|
rm -f "${pidfile}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# stop a single service (pidfile, then pgrep fallback)
|
||||||
|
function svc_stop() {
|
||||||
|
local name="$1" bin pid pidfile
|
||||||
|
bin="$(_svc_bin "$name")" || return 1
|
||||||
|
pidfile="$(_svc_pidfile "$name")"
|
||||||
|
|
||||||
|
pid="$(_svc_pid "$name")"
|
||||||
|
if [ -z "$pid" ]; then
|
||||||
|
# fallback: match by binary name
|
||||||
|
pid="$(pgrep -f "$(basename "$bin")" 2>/dev/null | head -n1)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -z "$pid" ]; then
|
||||||
|
_in "${name} is not running"
|
||||||
|
rm -f "${pidfile}"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
_in "Stopping ${name} (pid $pid)"
|
||||||
|
_svc_kill "$name" "" "$pid"
|
||||||
|
local i
|
||||||
|
for i in $(seq 1 10); do
|
||||||
|
_svc_alive "$name" "$pid" || break
|
||||||
|
sleep 0.3
|
||||||
|
done
|
||||||
|
if _svc_alive "$name" "$pid"; then
|
||||||
|
_in "${name} did not exit, sending SIGKILL"
|
||||||
|
_svc_kill "$name" 9 "$pid"
|
||||||
|
sleep 0.3
|
||||||
|
fi
|
||||||
|
rm -f "${pidfile}"
|
||||||
|
_in "${name} stopped"
|
||||||
|
}
|
||||||
|
|
||||||
|
# restart a single service
|
||||||
|
function svc_restart() {
|
||||||
|
local name="$1"
|
||||||
|
svc_stop "$name"
|
||||||
|
svc_start "$name"
|
||||||
|
}
|
||||||
|
|
||||||
|
# print status of a single service
|
||||||
|
function svc_status() {
|
||||||
|
local name="$1" bin pid pidfile log state
|
||||||
|
bin="$(_svc_bin "$name")" || return 1
|
||||||
|
pidfile="$(_svc_pidfile "$name")"
|
||||||
|
log="${g_log_dir}/${name}.log"
|
||||||
|
|
||||||
|
pid="$(_svc_pid "$name")"
|
||||||
|
if [ -n "$pid" ]; then
|
||||||
|
state="running"
|
||||||
|
else
|
||||||
|
state="stopped"
|
||||||
|
fi
|
||||||
|
printf "%-12s %-8s %-8s\n" "$name" "$state" "${pid:--}"
|
||||||
|
printf " binary: %s\n" "$bin"
|
||||||
|
if [ -f "$log" ]; then
|
||||||
|
printf " log: %s\n" "$log"
|
||||||
|
printf " last: %s\n" "$(tail -n1 "$log" 2>/dev/null)"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---- service management end ----
|
||||||
|
|
||||||
|
# clean
|
||||||
|
function build_clean() {
|
||||||
|
_in "Cleaning build artifacts..."
|
||||||
|
local src_dir=${g_code_base}/src
|
||||||
|
for dir in build build-prod; do
|
||||||
|
if [ -d "${src_dir}/${dir}" ]; then
|
||||||
|
run_cmd "rm -rf ${src_dir}/${dir}"
|
||||||
|
_in "Removed ${dir}/"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
function build_dev() {
|
||||||
|
_in "Building development version..."
|
||||||
|
local src_dir=${g_code_base}/src
|
||||||
|
cd ${src_dir}
|
||||||
|
local start_ts=$(date +%s)
|
||||||
|
run_cmd "nix develop --extra-experimental-features \"nix-command flakes\" --command bash -c \"meson setup build --buildtype=debug -Db_sanitize=address && meson compile -C build\""
|
||||||
|
local end_ts=$(date +%s)
|
||||||
|
local elapsed=$((end_ts - start_ts))
|
||||||
|
_in "Build finished in $((elapsed / 60))m $((elapsed % 60))s (${elapsed}s total)"
|
||||||
|
echo "output is in ${src_dir}/build"
|
||||||
|
}
|
||||||
|
|
||||||
|
function build_prod() {
|
||||||
|
_in "Building production version..."
|
||||||
|
local src_dir=${g_code_base}/src
|
||||||
|
cd ${src_dir}
|
||||||
|
local start_ts=$(date +%s)
|
||||||
|
run_cmd "nix develop --extra-experimental-features \"nix-command flakes\" --command bash -c \"meson setup build-prod --buildtype=release && meson compile -C build-prod\""
|
||||||
|
local end_ts=$(date +%s)
|
||||||
|
local elapsed=$((end_ts - start_ts))
|
||||||
|
_in "Build finished in $((elapsed / 60))m $((elapsed % 60))s (${elapsed}s total)"
|
||||||
|
echo "output is in ${src_dir}/build-prod"
|
||||||
|
}
|
||||||
|
|
||||||
|
# deploy cluster config: build cluster.bin from src/configs via xceadmin
|
||||||
|
# (in nix develop), place it in ctrl-1/conf, set conf/current, then restart
|
||||||
|
# the controller so it loads the config. Without a deployed cluster.bin the
|
||||||
|
# controller rejects worker joins with EINVAL (no hostconfig matched).
|
||||||
|
function config_deploy() {
|
||||||
|
_in "Deploying cluster config..."
|
||||||
|
local src_dir=${g_code_base}/src
|
||||||
|
local xceadmin=""
|
||||||
|
local candidate
|
||||||
|
|
||||||
|
# xceadmin lives under whichever meson build dir; try each in turn
|
||||||
|
for candidate in \
|
||||||
|
"${src_dir}/build/xceadmin/.venv/bin/xceadmin" \
|
||||||
|
"${src_dir}/build-prod/xceadmin/.venv/bin/xceadmin" \
|
||||||
|
"$(command -v xceadmin 2>/dev/null || true)"
|
||||||
|
do
|
||||||
|
if [ -n "$candidate" ] && [ -x "$candidate" ]; then
|
||||||
|
xceadmin="$candidate"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if [ -z "$xceadmin" ]; then
|
||||||
|
_in "xceadmin not found (run ./all.sh build first)"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
_in "Using xceadmin: ${xceadmin}"
|
||||||
|
|
||||||
|
local tmpbin="${g_run_dir}/.cluster.bin.tmp"
|
||||||
|
local confdir="${g_run_dir}/controller/ctrl-1/conf"
|
||||||
|
|
||||||
|
_in "Building cluster.bin from ${src_dir}/configs (in nix develop)"
|
||||||
|
cd ${src_dir}
|
||||||
|
run_cmd "nix develop --extra-experimental-features \"nix-command flakes\" --command bash -c \"${xceadmin} config build ${src_dir}/configs -o ${tmpbin}\""
|
||||||
|
|
||||||
|
if [ ! -s "${tmpbin}" ]; then
|
||||||
|
_in "cluster.bin build failed -- check output above"
|
||||||
|
rm -f "${tmpbin}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "${confdir}"
|
||||||
|
local tag
|
||||||
|
tag="$(sha256sum "${tmpbin}" | cut -c1-12)"
|
||||||
|
mv "${tmpbin}" "${confdir}/cluster.bin.${tag}"
|
||||||
|
echo "cluster.bin.${tag}" > "${confdir}/current"
|
||||||
|
_in "Deployed cluster.bin.${tag} (current -> cluster.bin.${tag}) in ${confdir}"
|
||||||
|
|
||||||
|
_in "Restarting controller to load config"
|
||||||
|
svc_restart controller
|
||||||
|
}
|
||||||
|
|
||||||
|
# back up all services' runtime configs (controller certs/conf, worker.yaml,
|
||||||
|
# apiserver configs) to ~/all_configs, preserving the directory structure.
|
||||||
|
function config_backup() {
|
||||||
|
_in "Backing up all service configs to ~/all_configs..."
|
||||||
|
local dest=~/all_configs
|
||||||
|
local n=0
|
||||||
|
|
||||||
|
# controller: TLS certs
|
||||||
|
if [ -d "${g_run_dir}/certs" ]; then
|
||||||
|
mkdir -p "${dest}/controller/certs"
|
||||||
|
cp -f ${g_run_dir}/certs/* "${dest}/controller/certs/" 2>/dev/null
|
||||||
|
_in "controller certs -> ${dest}/controller/certs/"
|
||||||
|
n=$((n + 1))
|
||||||
|
fi
|
||||||
|
|
||||||
|
# controller: cluster.bin + current
|
||||||
|
if [ -d "${g_run_dir}/controller/ctrl-1/conf" ]; then
|
||||||
|
mkdir -p "${dest}/controller/conf"
|
||||||
|
cp -f ${g_run_dir}/controller/ctrl-1/conf/cluster.bin.* \
|
||||||
|
"${g_run_dir}/controller/ctrl-1/conf/current" \
|
||||||
|
"${dest}/controller/conf/" 2>/dev/null
|
||||||
|
_in "controller conf -> ${dest}/controller/conf/"
|
||||||
|
n=$((n + 1))
|
||||||
|
fi
|
||||||
|
|
||||||
|
# worker: ${g_conf_dir}/worker.yaml (generated by svc_prepare_worker)
|
||||||
|
mkdir -p "${dest}/worker"
|
||||||
|
if [ -f "${g_conf_dir}/worker.yaml" ]; then
|
||||||
|
cp -f "${g_conf_dir}/worker.yaml" "${dest}/worker/"
|
||||||
|
_in "worker.yaml -> ${dest}/worker/"
|
||||||
|
n=$((n + 1))
|
||||||
|
else
|
||||||
|
_in "worker.yaml (${g_conf_dir}/worker.yaml) not found, skipped"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# apiserver: 4 config files
|
||||||
|
if [ -d "${g_run_dir}/apiserver/config" ]; then
|
||||||
|
mkdir -p "${dest}/apiserver"
|
||||||
|
cp -f ${g_run_dir}/apiserver/config/*.yaml "${dest}/apiserver/" 2>/dev/null
|
||||||
|
_in "apiserver config -> ${dest}/apiserver/"
|
||||||
|
n=$((n + 1))
|
||||||
|
fi
|
||||||
|
|
||||||
|
# scheduler: no standalone config file
|
||||||
|
mkdir -p "${dest}/scheduler"
|
||||||
|
printf 'scheduler 无独立配置文件（启动参数 -l <logfile> 已由 all.sh svc_args_scheduler 管理）\n' \
|
||||||
|
> "${dest}/scheduler/README.txt"
|
||||||
|
|
||||||
|
_in "Backup complete (${n} groups) in ${dest}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# main sub-command dispatch
|
||||||
|
function cmd_build() {
|
||||||
|
local mode="dev"
|
||||||
|
while getopts "cdp" opt; do
|
||||||
|
case $opt in
|
||||||
|
c) build_clean; exit 0 ;;
|
||||||
|
d) mode="dev" ;;
|
||||||
|
p) mode="prod" ;;
|
||||||
|
?) echo "Usage: $0 build [-c] [-d|-p]" && exit 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "$mode" = "dev" ]; then
|
||||||
|
build_dev
|
||||||
|
else
|
||||||
|
build_prod
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
function _cmd_svc_foreach() {
|
||||||
|
# $1 = action (start|stop|restart|status), rest = services
|
||||||
|
local action="$1"; shift
|
||||||
|
local targets rc=0
|
||||||
|
targets="$(_svc_resolve "$@")" || exit 1
|
||||||
|
local name
|
||||||
|
for name in $targets; do
|
||||||
|
if ! "svc_${action}" "$name"; then
|
||||||
|
rc=1
|
||||||
|
# fail-fast for start/restart: one bad service (e.g. controller
|
||||||
|
# without cluster.bin) should stop the whole launch so the user
|
||||||
|
# sees the precheck hint (./all.sh config deploy) immediately.
|
||||||
|
case "$action" in
|
||||||
|
start|restart) break ;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
return "$rc"
|
||||||
|
}
|
||||||
|
|
||||||
|
function cmd_start() { _cmd_svc_foreach start "$@"; }
|
||||||
|
function cmd_stop() { _cmd_svc_foreach stop "$@"; }
|
||||||
|
function cmd_restart() { _cmd_svc_foreach restart "$@"; }
|
||||||
|
|
||||||
|
function cmd_status() {
|
||||||
|
local targets
|
||||||
|
targets="$(_svc_resolve "$@")" || return 1
|
||||||
|
printf "%-12s %-8s %-8s\n" "SERVICE" "STATE" "PID"
|
||||||
|
local name
|
||||||
|
for name in $targets; do
|
||||||
|
svc_status "$name"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
function cmd_config() {
|
||||||
|
case "${1:-deploy}" in
|
||||||
|
deploy) config_deploy ;;
|
||||||
|
backup) config_backup ;;
|
||||||
|
*) echo "Usage: $0 config {deploy|backup}" && exit 1 ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# dispatch
|
||||||
|
case "${1:-}" in
|
||||||
|
build) shift; cmd_build "$@" ;;
|
||||||
|
config) shift; cmd_config "$@" ;;
|
||||||
|
start) shift; cmd_start "$@" ;;
|
||||||
|
stop) shift; cmd_stop "$@" ;;
|
||||||
|
restart) shift; cmd_restart "$@" ;;
|
||||||
|
status) shift; cmd_status "$@" ;;
|
||||||
|
*) echo "Usage: $0 {build|config|start|stop|restart|status} [args]" && exit 1 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
595
all_README.md
Normal file
595
all_README.md
Normal file
|
|
@ -0,0 +1,595 @@
|
||||||
|
# all.sh 使用与实现说明
|
||||||
|
|
||||||
|
本文件记录 `all.sh` 的功能、使用方法和内部实现细节，便于后续维护与扩展。
|
||||||
|
|
||||||
|
## 1. 概述
|
||||||
|
|
||||||
|
`all.sh` 是 XCE 项目的统一管理脚本，提供两个能力：
|
||||||
|
|
||||||
|
1. **构建**：以 `nix develop` + `meson` 构建开发版 / 生产版 / 清理构建产物，并输出编译耗时统计。
|
||||||
|
2. **服务管理**：启动 / 停止 / 重启 / 查看状态四个核心程序：
|
||||||
|
- `xce_controller`（controller，控制器，默认端口 9990）
|
||||||
|
- `xce_scheduler`（scheduler，调度器，默认连接 localhost:9990）
|
||||||
|
- `xcecd`（worker，工作节点守护进程，需要 root）
|
||||||
|
- `xce_apiserver`（apiserver，REST API，默认 HTTP 端口 8080，连接 controller）
|
||||||
|
|
||||||
|
所有日志统一输出到 `logs/`，PID 统一记录在 `run/`。
|
||||||
|
|
||||||
|
## 2. 快速使用
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# ---------- 构建 ----------
|
||||||
|
./all.sh build -c # 彻底清理构建产物（删除 build/ 与 build-prod/）
|
||||||
|
./all.sh build -d # 开发构建（debug + ASan，带编译耗时统计）
|
||||||
|
./all.sh build -p # 生产构建（release，无 sanitizer，带编译耗时统计）
|
||||||
|
|
||||||
|
# ---------- 集群配置 ----------
|
||||||
|
./all.sh config deploy # 构建+部署 cluster.bin（从 src/configs）并重启 controller
|
||||||
|
./all.sh config backup # 备份所有服务配置到 ~/all_configs
|
||||||
|
|
||||||
|
# ---------- 服务管理 ----------
|
||||||
|
./all.sh start # 启动全部服务（controller scheduler worker apiserver）
|
||||||
|
./all.sh start controller scheduler # 只启动指定服务
|
||||||
|
./all.sh stop [svc...] # 停止指定服务，缺省为全部
|
||||||
|
./all.sh restart [svc...] # 重启指定服务，缺省为全部
|
||||||
|
./all.sh status [svc...] # 查看状态（名称 / 状态 / PID / 二进制 / 日志），缺省为全部
|
||||||
|
```
|
||||||
|
|
||||||
|
服务清单（`svc` 可取值）：`controller` / `scheduler` / `worker` / `apiserver`。
|
||||||
|
|
||||||
|
## 3. 配置区（customize 区）
|
||||||
|
|
||||||
|
脚本顶部 `# customize begin ... # customize end` 是可定制区域：
|
||||||
|
|
||||||
|
| 变量 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| `g_code_base` | 项目根目录，默认 `/home/jsxing/allwork/git/myxce` |
|
||||||
|
| `g_services` | 服务注册表：`名称|二进制绝对路径` 数组 |
|
||||||
|
| `g_deploy_dir` | 统一部署根目录，默认 `${g_code_base}/deploy`（所有配置/logs/run 都收在这里） |
|
||||||
|
| `g_conf_dir` | 生成配置目录，默认 `${g_deploy_dir}/conf`（worker.yaml 等） |
|
||||||
|
| `g_run_dir` | PID/证书/数据目录，默认 `${g_deploy_dir}/run` |
|
||||||
|
| `g_log_dir` | 日志目录，默认 `${g_deploy_dir}/logs` |
|
||||||
|
| `g_loglevel` | 全局日志级别：`info` / `debug` / `debug2` / `debug3`，默认 `info` |
|
||||||
|
| `g_worker_bin_dir` | 作业 worker 二进制（`xcecd_worker`）安装目录，默认 `/opt/xce/bin`；必须选一个所有作业执行用户都能到达的位置（仓库路径在 `/home/<user>` 下通常 750，其他用户无法进入，作业会以 exit 126 立即退出，见 9.4）。由 `svc_prepare_worker` 经 sudo 安装、`svc_env_worker` 以 `XCE_SERVER_DIR` 注入 |
|
||||||
|
|
||||||
|
### 3.1 日志级别 `g_loglevel`
|
||||||
|
|
||||||
|
一个开关控制所有组件的日志详细度。修改后 **restart 对应服务** 即生效（配置会在启动前自动同步）：
|
||||||
|
|
||||||
|
| `g_loglevel` | controller | scheduler | worker (xcecd) | apiserver |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `info`（默认） | 无 flag | 无 flag | worker.yaml `log_level: "info"` | logger.yaml `level: "info"` |
|
||||||
|
| `debug` | `-v` | `-v` | `log_level: "debug"` | `level: "debug"` |
|
||||||
|
| `debug2` | `-vv` | `-vv` | `log_level: "debug2"` | `level: "debug"`（zap 无 debug2） |
|
||||||
|
| `debug3` | `-vvv` | `-vvv` | `log_level: "debug3"` | `level: "debug"` |
|
||||||
|
|
||||||
|
实现要点：
|
||||||
|
- **controller / scheduler**：`svc_args_*` 通过 `$(_verbose_flags)` 注入 `-v`/`-vv`/`-vvv`（argp `--verbose`，`verbose_to_loglevel()` 映射到 debug/debug2/debug3）。
|
||||||
|
- **worker**：`svc_prepare_worker` 把 `$(_worker_loglevel)` 写进 `${g_conf_dir}/worker.yaml` 的 `log_level` 字段。注意 xcecd 传了 `-c` 后会**忽略 `-l/-p/-m`**，日志级别/目录完全由 worker.yaml 决定；文件已存在时也会 `sed` 同步级别。
|
||||||
|
- **apiserver**：`svc_prepare_apiserver` 把 `$(_apiserver_loglevel)` 写进 `<topdir>/config/logger.yaml` 的 `level`（zap 只支持 `debug/info/warn/error/fatal`，故 debug2/3 映射为 `debug`）；文件已存在时同样同步。
|
||||||
|
|
||||||
|
### 服务注册表
|
||||||
|
|
||||||
|
```bash
|
||||||
|
g_services=(
|
||||||
|
"controller|${g_code_base}/src/build/controller/xce_controller"
|
||||||
|
"scheduler|${g_code_base}/src/build/scheduler/xce_scheduler"
|
||||||
|
"worker|${g_code_base}/src/build/worker/xcecd"
|
||||||
|
"apiserver|${g_code_base}/src/build/frontend/apiserver/xce_apiserver"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
新增服务只需在此追加一行 `"名字|二进制路径"`，并补上对应的 `svc_args_<名字>` 函数（见下）。
|
||||||
|
|
||||||
|
## 4. 构建实现
|
||||||
|
|
||||||
|
构建统一使用：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nix develop --extra-experimental-features "nix-command flakes" \
|
||||||
|
--command bash -c "meson setup <build目录> <参数> && meson compile -C <build目录>"
|
||||||
|
```
|
||||||
|
|
||||||
|
> 注意：`meson setup` 和 `meson compile` 必须都跑在 `nix develop` 环境内，因此被放进同一个 `--command bash -c`。
|
||||||
|
|
||||||
|
### 4.1 build -d（开发版）
|
||||||
|
|
||||||
|
- 构建目录：`src/build/`
|
||||||
|
- 参数：`--buildtype=debug -Db_sanitize=address`（调试符号 + AddressSanitizer）
|
||||||
|
- 输出：`output is in ${src_dir}/build`
|
||||||
|
|
||||||
|
### 4.2 build -p（生产版）
|
||||||
|
|
||||||
|
- 构建目录：`src/build-prod/`
|
||||||
|
- 参数：`--buildtype=release`（开启优化，无 sanitizer）
|
||||||
|
- 输出：`output is in ${src_dir}/build-prod`
|
||||||
|
|
||||||
|
### 4.3 build -c（清理）
|
||||||
|
|
||||||
|
- 循环删除 `src/build/` 与 `src/build-prod/` 两个目录，彻底清空 meson 产物。
|
||||||
|
|
||||||
|
### 4.4 编译耗时统计
|
||||||
|
|
||||||
|
用 `date +%s` 在 `meson compile` 前后各取一次时间戳，结束时打印：
|
||||||
|
|
||||||
|
```
|
||||||
|
Build finished in 0m 56s (56s total)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.5 构建耗时经验值
|
||||||
|
|
||||||
|
- `-d/-p`：增量编译，只有 xceadmin 每次固定重跑（约 11s）+ 首次/源码变更时全量。初次全量约 60s。
|
||||||
|
- `-c`：彻底清理后需要完整重建。
|
||||||
|
|
||||||
|
### 4.6 config deploy（部署集群配置）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./all.sh config deploy
|
||||||
|
```
|
||||||
|
|
||||||
|
一条命令完成：
|
||||||
|
1. 定位 `xceadmin`（依次查找 `src/build`、`src/build-prod` 的 venv、PATH）。
|
||||||
|
2. 在 `nix develop` 环境中用 `xceadmin config build src/configs -o ...` 构建 `cluster.bin`（schema 校验会输出若干 "field ... ignored" 警告，属正常）。
|
||||||
|
3. 按 sha256 前 12 位生成 tag，部署到 `run/controller/ctrl-1/conf/cluster.bin.<tag>`，写入 `conf/current`。
|
||||||
|
4. 重启 controller 加载配置（日志见 `Primary config loaded: tag=...`，`ctrlstatus` 显示 `Cluster: <name>`）。
|
||||||
|
|
||||||
|
> 为什么要部署：无 cluster.bin 时 controller 处于冷启动无配置状态，**worker join 会被拒绝并返回 EINVAL**（`find_matched_hostconfig` 因 `cluster_config == NULL` 返回 NULL，见 8 节）。清空重建（`build -c`）或删掉 `run/` 后必须重新 `config deploy`。
|
||||||
|
|
||||||
|
### 4.7 config backup（备份所有服务配置）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./all.sh config backup
|
||||||
|
```
|
||||||
|
|
||||||
|
把全部服务的运行时配置备份到 `~/all_configs/`（保留目录结构，覆盖式备份）：
|
||||||
|
|
||||||
|
```
|
||||||
|
~/all_configs/
|
||||||
|
├── controller/certs/ # TLS 证书 xce.crt / xce.key
|
||||||
|
├── controller/conf/ # cluster.bin.* + current
|
||||||
|
├── worker/worker.yaml # /opt/xce/conf/worker.yaml（sudo 读取）
|
||||||
|
├── apiserver/ # app.yaml / auth.yaml / logger.yaml / rbac.yaml
|
||||||
|
└── scheduler/README.txt # 说明：scheduler 无独立配置文件
|
||||||
|
```
|
||||||
|
|
||||||
|
缺失的文件会被跳过并提示（例如未启动过 apiserver 时无其配置目录）。
|
||||||
|
|
||||||
|
## 5. 服务管理实现
|
||||||
|
|
||||||
|
服务管理采用「注册表 + 每服务钩子函数」的可扩展设计。
|
||||||
|
|
||||||
|
### 5.1 每服务启动参数函数 `svc_args_<name>`
|
||||||
|
|
||||||
|
`svc_start` 通过 `svc_args_${name}` 动态调用，返回该服务的启动参数。
|
||||||
|
|
||||||
|
| 服务 | 参数 |
|
||||||
|
|------|------|
|
||||||
|
| `svc_args_controller` | `--myname ctrl-1 --peers ctrl-1=localhost:9990 --workdir <deploy>/run/controller --certdir <deploy>/run/certs --logfile <deploy>/logs/controller.log --vcpus 4 [<g_loglevel>]` |
|
||||||
|
| `svc_args_scheduler` | `-l <deploy>/logs/scheduler.log [<g_loglevel>]` |
|
||||||
|
| `svc_args_worker` | `-c <deploy>/conf/worker.yaml -n localhost01`（xcecd 传 `-c` 后忽略 `-l/-p/-m`，日志位置/级别由 worker.yaml 决定） |
|
||||||
|
| `svc_args_apiserver` | `-d <deploy>/run/apiserver` |
|
||||||
|
|
||||||
|
> 表中 `[<g_loglevel>]` 表示按 `g_loglevel` 注入的 `-v`/`-vv`/`-vvv`（见 3.1 节）。
|
||||||
|
|
||||||
|
另外还有每服务环境变量钩子 `svc_env_<name>`（可选）：`svc_start` 在 sudo 启动分支里用 `sudo -n env <值> bash -c ...` 注入到进程环境。
|
||||||
|
|
||||||
|
| 钩子 | 注入的环境变量 | 说明 |
|
||||||
|
|------|--------------|------|
|
||||||
|
| `svc_env_worker` | `XCE_SERVER_DIR=${g_worker_bin_dir}` | 让 `xcecd` 用安装在 `${g_worker_bin_dir}`（默认 `/opt/xce/bin`）的 `xcecd_worker` 启动作业 worker，而不是仓库路径；否则作业会以 exit 126 立即退出（见 9.4） |
|
||||||
|
|
||||||
|
### 5.2 启动前准备钩子 `svc_prepare_<name>`
|
||||||
|
|
||||||
|
`svc_start` 在启动前检查 `svc_prepare_${name}` 是否存在，存在则先执行（如失败则中止启动）。
|
||||||
|
|
||||||
|
- `svc_prepare_controller`：确保自签名 TLS 证书存在。若 `<deploy>/run/certs/xce.crt` 或 `xce.key` 缺失，用 `openssl` 生成（RSA 2048、10 年有效期）。controller 缺少证书会启动失败（`TLS certificate not found`）。
|
||||||
|
|
||||||
|
- `svc_prepare_worker`：生成 `${g_conf_dir}/worker.yaml`（`management_list`/`controller_port`/`log_level`/`log_dir`），其中 `log_level` 跟随 `g_loglevel`、`log_dir` 指向 `${g_log_dir}`。文件已存在时也会 `sed` 同步 `log_level`，因此修改 `g_loglevel` 后重启 worker 即生效。
|
||||||
|
- 另外会把 `src/build/worker/xcecd_worker` 用 `sudo install -m 0755` 安装到 `${g_worker_bin_dir}`（默认 `/opt/xce/bin`），源更新才重装（幂等）。作业 worker 由 systemd 以**作业执行用户**身份运行，必须放在所有人可访问的目录（仓库路径在 `/home/<user>` 下通常 750，其他用户无法进入 → exit 126，见 9.4）。
|
||||||
|
|
||||||
|
- `svc_prepare_apiserver`：确保 `<deploy>/run/apiserver/config/` 下的四个配置文件齐全（缺失才生成，不会覆盖已有配置）：
|
||||||
|
- `logger.yaml`：日志配置（相对 `outputPath` 会落在 `<topdir>/logs/`），`level` 跟随 `g_loglevel`（zap 只支持 debug/info/warn/error，debug2/3 映射为 debug）；已存在时也会同步 `level`。
|
||||||
|
- `rbac.yaml`：权限配置；优先复制仓库样例 `src/frontend/apiserver/config/rbac.yaml`，否则生成最小空配置。
|
||||||
|
- `auth.yaml`：认证配置；优先复制仓库样例 `src/frontend/apiserver/config/auth.yaml`（`config_file` 提供者，默认 admin/admin），否则生成最小单用户配置。
|
||||||
|
- `app.yaml`：主配置（`app`/`security`/`authentication`/`rate_limit`/`whitelist`/`file_upload`/`cache` 为 schema 必填），生成默认值并将 `backend.address/port` 指向 controller（`localhost:9990`）。
|
||||||
|
> 这四份文件缺一不可，Go 二进制启动时会按顺序校验（缺失即报错退出）。
|
||||||
|
|
||||||
|
### 5.3 sudo 支持 `svc_needs_sudo`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
function svc_needs_sudo() {
|
||||||
|
case "$1" in
|
||||||
|
worker) return 0 ;; # 需要 root
|
||||||
|
*) return 1 ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`xcecd` 校验 `geteuid() != 0` 直接退出（`FATAL: xcecd should be run as root`），因此 worker 必须 root 运行。
|
||||||
|
|
||||||
|
#### 启动（sudo 分支）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -n env ${envp} bash -c "echo \$\$ > '${pidfile}'; exec nohup '${bin}' ${args}" \
|
||||||
|
>>"${log}" 2>&1 < /dev/null &
|
||||||
|
```
|
||||||
|
|
||||||
|
`envp` 来自可选的 `svc_env_<name>` 钩子（如 worker 注入 `XCE_SERVER_DIR=${g_worker_bin_dir}`，见 5.1）；用 `env` 显式设置，避免受 sudo `env_reset` 影响。
|
||||||
|
|
||||||
|
设计要点：
|
||||||
|
- 内层 bash 先把**自己的 PID** 写入 pid 文件，再 `exec` 进二进制。`exec` 不改变 PID，因此 pid 文件记录的是**真实的 root 进程 PID**（而不是 sudo 的 PID）。
|
||||||
|
- `nohup` 防护 SIGHUP，避免终端关闭时被杀。
|
||||||
|
- `sudo -n` 非交互式：要求 sudo 免密（NOPASSWD）或已有缓存凭证（默认缓存 15 分钟），避免后台进程挂起等密码。
|
||||||
|
|
||||||
|
#### 状态检测与停止（root 进程）
|
||||||
|
|
||||||
|
普通用户对 root 进程执行 `kill`/`kill -0` 会返回 EPERM（失败），因此对 root 服务统一走 sudo：
|
||||||
|
|
||||||
|
- `_svc_alive`：`sudo -n kill -0 <pid>` 检测存活。
|
||||||
|
- `_svc_kill`：`sudo -n kill [-9] <pid>` 发信号。
|
||||||
|
|
||||||
|
### 5.4 启动流程 `svc_start`
|
||||||
|
|
||||||
|
1. 解析二进制路径，检查二进制存在（`-x`）。
|
||||||
|
2. 防重复启动：`_svc_pid` 已检测到存活 PID 则直接返回。
|
||||||
|
3. 创建 `run/`、`logs/` 目录。
|
||||||
|
4. 执行 `svc_prepare_<name>`（若存在）。
|
||||||
|
5. 执行 `svc_check_<name>` 预检（若存在，如 controller 需要已部署 cluster.bin）。
|
||||||
|
6. 按是否 sudo 分支后台启动，写 pid 文件，`disown`：
|
||||||
|
- sudo 分支先取 `svc_env_<name>` 注入的环境变量（若存在），再 `sudo -n env ... bash -c ...`。
|
||||||
|
- 非 sudo 分支 `setsid nohup <bin> <args>` 后台启动。
|
||||||
|
7. `sleep 1` 后通过 `_svc_pid` 校验：存活则报 `started (pid ...)`；否则报 `FAILED to start` 并清理 pid 文件。
|
||||||
|
|
||||||
|
### 5.5 停止流程 `svc_stop`
|
||||||
|
|
||||||
|
1. 从 pid 文件读 PID；若无，用 `pgrep -f <二进制名>` 兜底。
|
||||||
|
2. 发送 SIGTERM（root 服务走 sudo）。
|
||||||
|
3. 最多等待 3 秒（10 次 × 0.3s），仍存活则 SIGKILL。
|
||||||
|
4. 清理 pid 文件，报 `stopped`。
|
||||||
|
|
||||||
|
> 实测：worker 在 SIGTERM 后 3 秒内不退出，会走 SIGKILL 兜底，属预期行为。
|
||||||
|
|
||||||
|
### 5.6 状态显示 `svc_status`
|
||||||
|
|
||||||
|
```
|
||||||
|
SERVICE STATE PID
|
||||||
|
controller running 200831
|
||||||
|
binary: /home/jsxing/allwork/git/xce/src/build/controller/xce_controller
|
||||||
|
log: /home/jsxing/allwork/git/xce/logs/controller.log
|
||||||
|
last: <日志最后一行>
|
||||||
|
```
|
||||||
|
|
||||||
|
- `STATE` 由 `_svc_pid`（pid 文件 + 存活校验）决定，stale pid 文件会被自动清理。
|
||||||
|
- 显示日志文件路径及其最后一行，便于快速排查。
|
||||||
|
|
||||||
|
### 5.7 参数解析与分发
|
||||||
|
|
||||||
|
入口处的 `case` 把子命令分发给各 `cmd_*` 函数：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
case "${1:-}" in
|
||||||
|
build) shift; cmd_build "$@" ;;
|
||||||
|
config) shift; cmd_config "$@" ;;
|
||||||
|
start) shift; cmd_start "$@" ;;
|
||||||
|
stop) shift; cmd_stop "$@" ;;
|
||||||
|
restart) shift; cmd_restart "$@" ;;
|
||||||
|
status) shift; cmd_status "$@" ;;
|
||||||
|
*) echo "Usage: $0 {build|config|start|stop|restart|status} [args]" && exit 1 ;;
|
||||||
|
esac
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 服务列表解析 `_svc_resolve`
|
||||||
|
|
||||||
|
把命令行参数解析成服务名列表：缺省返回**全部**注册服务（`_svc_names`）；未知服务名会报错并提示合法值（错误写到 stderr，避免被命令替换吞掉返回值）。
|
||||||
|
|
||||||
|
#### 批量分发 `_cmd_svc_foreach`
|
||||||
|
|
||||||
|
`start`/`stop`/`restart` 三个子命令都经由它按服务逐个执行；`status` 不走这里（见下）。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
function _cmd_svc_foreach() {
|
||||||
|
local action="$1"; shift # start | stop | restart
|
||||||
|
local targets rc=0
|
||||||
|
targets="$(_svc_resolve "$@")" || exit 1
|
||||||
|
local name
|
||||||
|
for name in $targets; do
|
||||||
|
if ! "svc_${action}" "$name"; then
|
||||||
|
rc=1
|
||||||
|
# fail-fast for start/restart: one bad service (e.g. controller
|
||||||
|
# without cluster.bin) should stop the whole launch so the user
|
||||||
|
# sees the precheck hint (./all.sh config deploy) immediately.
|
||||||
|
case "$action" in
|
||||||
|
start|restart) break ;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
return "$rc"
|
||||||
|
}
|
||||||
|
|
||||||
|
function cmd_start() { _cmd_svc_foreach start "$@"; }
|
||||||
|
function cmd_stop() { _cmd_svc_foreach stop "$@"; }
|
||||||
|
function cmd_restart() { _cmd_svc_foreach restart "$@"; }
|
||||||
|
```
|
||||||
|
|
||||||
|
要点：
|
||||||
|
- **动态调用**：`"svc_${action}" "$name"` 即 `svc_start` / `svc_stop` / `svc_restart`，天然支持一次操作多个服务（如 `./all.sh start controller scheduler`）。
|
||||||
|
- **fail-fast**：`start`/`restart` 遇到某个服务失败立即 `break`（如 controller 未部署 cluster.bin 时 `svc_check_controller` 失败），让用户马上看到 `./all.sh config deploy` 的提示；`stop` 不打断，尽力停止其余服务。
|
||||||
|
- **退出码**：任何服务失败都累计 `rc=1`，脚本整体返回非 0。
|
||||||
|
- **`status` 不走 foreach**：`cmd_status` 自行循环调用 `svc_status`，先打印 `SERVICE/STATE/PID` 表头。
|
||||||
|
- **`config deploy` 不走 foreach**：`config_deploy` 内部直接 `svc_restart controller`。
|
||||||
|
|
||||||
|
## 6. 配置文件与运行时目录
|
||||||
|
|
||||||
|
> 所有配置、日志、运行时状态统一在 `${g_deploy_dir}`（`deploy/`）下，不再引用系统目录或旧仓库路径。
|
||||||
|
|
||||||
|
### 6.1 controller
|
||||||
|
|
||||||
|
- 数据目录：`<deploy>/run/controller/ctrl-1/`（`data/` WAL、`conf/` 配置），由 controller 自动创建。
|
||||||
|
- TLS 证书：`<deploy>/run/certs/`，由 `svc_prepare_controller` 自动生成。
|
||||||
|
- 日志：`<deploy>/logs/controller.log`。
|
||||||
|
|
||||||
|
### 6.2 worker 配置 `${g_conf_dir}/worker.yaml`
|
||||||
|
|
||||||
|
`xcecd` 的默认配置路径是 `DEFAULT_CONFIG_PATH`（`/opt/xce/conf/worker.yaml`），但 all.sh 始终用 `-c ${g_conf_dir}/worker.yaml` 显式指定，由 `svc_prepare_worker` 自动生成：
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
workerconf:
|
||||||
|
management_list: ["localhost"]
|
||||||
|
controller_port: 9990
|
||||||
|
log_level: "info"
|
||||||
|
log_dir: "/home/jsxing/allwork/git/myxce/deploy/logs"
|
||||||
|
```
|
||||||
|
|
||||||
|
- 样例参考：`src/worker/conf/worker.yaml`。
|
||||||
|
- 必填字段：`controller_port`（>0）、`log_dir`（非空）、`management_list`。
|
||||||
|
- 注意：传了 `-c` 后，`-l`/`-p`/`-m` 参数会被忽略（`main.c` 行为），因此 `log_dir`/`log_level` 必须写在配置里；`log_level` 跟随 `g_loglevel`（见 3.1）。
|
||||||
|
|
||||||
|
### 6.3 apiserver
|
||||||
|
|
||||||
|
- top 目录：`<deploy>/run/apiserver/`，启动参数 `-d <topdir>`（必填）。
|
||||||
|
- 配置文件：`<deploy>/run/apiserver/config/` 下需有 `app.yaml`、`logger.yaml`、`rbac.yaml`、`auth.yaml`（由 `svc_prepare_apiserver` 自动生成，见 5.2）。
|
||||||
|
- 后端连接：`app.yaml` 的 `backend` 段配置 controller 地址/端口（`localhost:9990`）；若后端未配置则运行在 stub 模式。
|
||||||
|
- HTTP 端口：`app.yaml` 的 `app.http_port`（默认 8080）。
|
||||||
|
- 日志：写在自己目录 `<deploy>/run/apiserver/logs/apiserver.log`（由 logger.yaml 决定），`<deploy>/logs/apiserver.log` 仅收集启动期输出。
|
||||||
|
- 默认账号：`admin` / `admin`（来自 `auth.yaml` 样例）。
|
||||||
|
|
||||||
|
## 7. xcecli 使用（登录与查询）
|
||||||
|
|
||||||
|
`xcecli` 是访问 apiserver（REST API，端口 8080）的命令行客户端，构建产物在 `src/build/frontend/xcecli/xcecli`（root 命令名为 `xce`）。
|
||||||
|
|
||||||
|
### 7.1 登录
|
||||||
|
|
||||||
|
```bash
|
||||||
|
src/build/frontend/xcecli/xcecli login -u admin -p admin --url http://localhost:8080
|
||||||
|
```
|
||||||
|
|
||||||
|
| 参数 | 必填 | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| `-u, --username` | ✅ | 用户名 |
|
||||||
|
| `-p, --password` | 可选 | 密码；不填则交互式提示输入（不回显） |
|
||||||
|
| `--url` | ✅ | API Server 地址，`http://IP:8080` 或 `https://IP:8443` |
|
||||||
|
|
||||||
|
- 成功输出 `login successfully`，令牌（JWT）保存到 `~/.xce/config.json`。
|
||||||
|
- 默认账号定义在 `run/apiserver/config/auth.yaml`：`admin`/`admin`、`rshen`/`Letmein`（cluster_admin）、`user1`/`user1`、`user2`/`user2`（basic_user）。
|
||||||
|
|
||||||
|
### 7.2 登录后的常用命令
|
||||||
|
|
||||||
|
```
|
||||||
|
xce login # 登录
|
||||||
|
xce logout # 退出（清除该服务器登录信息）
|
||||||
|
xce list # 列出已登录的 API Server
|
||||||
|
xce id # 查询集群信息（无集群配置时返回：no cluster found）
|
||||||
|
xce hosts list # 查询主机（无主机时返回：no host found）
|
||||||
|
xce jobs ... # 作业管理
|
||||||
|
xce queues ... # 队列管理
|
||||||
|
xce users ... # 用户管理
|
||||||
|
xce roles ... # 角色管理
|
||||||
|
xce permissions # 权限管理
|
||||||
|
xce resources # 资源管理
|
||||||
|
```
|
||||||
|
|
||||||
|
> 注：单节点冷启动未配置 `cluster.bin` 时，集群/主机等查询返回"无数据"属预期行为，认证链路本身已验证正常。
|
||||||
|
|
||||||
|
## 8. xce_test 使用（直连 controller 的测试客户端）
|
||||||
|
|
||||||
|
`xce_test` 是直接连接 `xce_controller`（端口 9990，走 TLS 协议）的测试客户端，**绕过 apiserver**，直接读写集群状态。构建产物在 `src/build/controller/xce_test`（与 `xce_controller` 一同构建）。
|
||||||
|
|
||||||
|
> 与 `xcecli` 的区别：`xcecli` 走 apiserver 的 HTTP REST API（8080）；`xce_test` 直连 controller 本身（9990），是 HA soak 测试（`/xce-test-ha`、`run-ha-soak.sh`）的客户端驱动。
|
||||||
|
|
||||||
|
```
|
||||||
|
jsxing@test01:~$ xce_test
|
||||||
|
xce>client
|
||||||
|
xce/client>
|
||||||
|
xce/client>workerlist
|
||||||
|
HostName Status HostType HostModel CPUfactor MaxCPU MaxMem MaxSwap MaxTmp NDisks
|
||||||
|
test01 UNAVAIL unknown unknown 0.000000 0 0.000000 0.000000 0.000000 0
|
||||||
|
h1 UNAVAIL unknown unknown 0.000000 0 0.000000 0.000000 0.000000 0
|
||||||
|
```
|
||||||
|
|
||||||
|
### 8.1 用法（stdin 命令脚本）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
printf 'set masters ctrl-1=localhost:9990
|
||||||
|
set myname probe
|
||||||
|
client
|
||||||
|
wait-primary 3000
|
||||||
|
ctrlstatus
|
||||||
|
quit
|
||||||
|
' | ASAN_OPTIONS=detect_leaks=0 src/build/controller/xce_test
|
||||||
|
```
|
||||||
|
|
||||||
|
实测输出：
|
||||||
|
```
|
||||||
|
wait-primary: primary 'ctrl-1' serving after 48 ms
|
||||||
|
Primary: ctrl-1
|
||||||
|
term: 4
|
||||||
|
committedLsn: 0
|
||||||
|
appliedLsn: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
> `ASAN_OPTIONS=detect_leaks=0` 是必需的：debug/ASAN 构建在退出时会报无害的内存泄漏，污染退出码（但不影响命令输出）。
|
||||||
|
|
||||||
|
### 8.2 常用设置
|
||||||
|
|
||||||
|
| 命令 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| `set masters ctrl-1=localhost:9990` | controller 候选列表（支持 HA failover，多个用逗号分隔，格式同 `--peers`） |
|
||||||
|
| `set myname probe` | 客户端标识 |
|
||||||
|
| `set secret ...` | 认证密钥（默认 `keepSecret`） |
|
||||||
|
|
||||||
|
### 8.3 命令结构（三个上下文）
|
||||||
|
|
||||||
|
| 上下文 | 进入命令 | 可用命令 |
|
||||||
|
|--------|---------|---------|
|
||||||
|
| **client**（客户端） | `client` | `wait-primary`、`ctrlstatus`、`jobsub`/`jobrun`/`jobkill`/`jobckpt`/`jobinfo`/`joblist`、`workerlist`、`reslist`/`resval`、`qctrl`、`hello`、`expect-job(s)`、`shutdown`、`taskrun`、`waldump`/`walcompare`/`snapverify`/`snapdiff` |
|
||||||
|
| **worker**（模拟工作节点） | `worker` | `join`、`jstart`/`jfinish`/`jfail`、`resupdate` |
|
||||||
|
| **scheduler**（模拟调度器） | `scheduler` | `join`、`getconf`、`setpend` |
|
||||||
|
|
||||||
|
> 注：单节点冷启动未配置 `cluster.bin` 时，`workerlist`/`joblist`/`reslist` 等返回空属预期行为；连接与查询链路本身已验证正常。
|
||||||
|
|
||||||
|
## 9. 修复的问题记录
|
||||||
|
|
||||||
|
### 9.1 controller 在 scheduler 加入时 ASan 崩溃（已修复）
|
||||||
|
|
||||||
|
**现象**：单节点冷启动（无 `cluster.bin` 配置）时启动 scheduler，controller 立即崩溃：
|
||||||
|
|
||||||
|
```
|
||||||
|
AddressSanitizer: SEGV on unknown address 0x000000000038
|
||||||
|
#0 xcesched_push_configs ../controller/sched.c:172
|
||||||
|
#1 on_scheduler_join ../controller/sched.c:667
|
||||||
|
#2 handle_scheduler_join ../controller/join.c:165
|
||||||
|
```
|
||||||
|
|
||||||
|
**根因**：冷启动时 `c->myconf->cluster_config` 为 NULL，`xcesched_push_configs` 无条件解引用（空指针 + 偏移 0x38）。同文件 `worker.c` 的 `find_matched_hostconfig` 有防护，唯独 `sched.c` 没有。
|
||||||
|
|
||||||
|
**修复**（`src/controller/sched.c`）：
|
||||||
|
|
||||||
|
```c
|
||||||
|
if (c->myconf == NULL || c->myconf->cluster_config == NULL) {
|
||||||
|
WARN("%s: cluster_config not loaded yet, skipping config push", __func__);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
配置未就绪时跳过推送、不崩溃，scheduler 之后重新加入时会收到配置。
|
||||||
|
|
||||||
|
**验证**：修复后 controller 日志出现 `cluster_config not loaded yet, skipping config push`，scheduler 成功接入，进程稳定。
|
||||||
|
|
||||||
|
### 9.2 单节点冷启动时序（正常行为）
|
||||||
|
|
||||||
|
controller 持久化了 `term` 后重启，会先进入 `CTRL_STANDBY`：
|
||||||
|
|
||||||
|
1. 启动时 `load_primary_config` 找不到配置 → "will fetch from primary after election"
|
||||||
|
2. 进入 `CTRL_STANDBY`，等待 primary
|
||||||
|
3. 约 10s liveness 超时 → 触发 PreVote failover
|
||||||
|
4. PreVote 单节点立即胜出 → 进入真实选举 → `CTRL_PRIMARY`（term+1）
|
||||||
|
5. 此期间 scheduler 的 role-2 join 会被拒绝（`standby has no primary yet`），scheduler 自动重试，成为 primary 后即接入
|
||||||
|
|
||||||
|
因此启动顺序无关紧要：scheduler / worker 连不上会重试，controller 竞选完成后自然接入。
|
||||||
|
|
||||||
|
### 9.3 worker join 返回 EINVAL（无集群配置）
|
||||||
|
|
||||||
|
**现象**：`xce_test` worker 上下文执行 `join` 报 `xce_join: Invalid argument`。
|
||||||
|
|
||||||
|
**根因**：与 9.1 同一个根因（冷启动无 `cluster.bin`）。`handle_worker_join`（join.c）先调 `find_matched_hostconfig` 找匹配主机配置，而该函数（worker.c）在 `c->myconf->cluster_config == NULL` 时返回 NULL 并设 `errno = EINVAL`，controller 据此回复 `-EINVAL`。
|
||||||
|
|
||||||
|
**修复**：部署集群配置后重启 controller——即 `./all.sh config deploy`。修复后 `join` 输出 `host join ok`，`workerlist`（client 上下文）可见 `h1` 已注册。
|
||||||
|
|
||||||
|
> 注意：`workerlist`/`joblist`/`reslist` 等查询命令只在 **client** 上下文可用，worker 上下文里没有这些命令（正常命令集划分）。
|
||||||
|
### 9.4 作业可调度但运行时立即退出（exit 126，Permission denied）
|
||||||
|
|
||||||
|
**现象**：作业能被调度（`jobs list` 可见），但一运行就结束。`xcecd.log` 显示：
|
||||||
|
|
||||||
|
```
|
||||||
|
Worker 188071 exited with code 126
|
||||||
|
```
|
||||||
|
|
||||||
|
systemd journal 给出真正原因：
|
||||||
|
|
||||||
|
```
|
||||||
|
sh[188071]: /bin/sh: 1: exec: /home/jsxing/allwork/git/myxce/src/build/worker/xcecd_worker: Permission denied
|
||||||
|
systemd[1]: xcecd-worker-1-0-...service: Main process exited, code=exited, status=126/n/a
|
||||||
|
```
|
||||||
|
|
||||||
|
**根因**：每个作业的 worker（`xcecd_worker`）由 systemd 临时单元以**作业执行用户**（如 `admin`）身份启动（`User=<execuser>`）。而 `xcecd` 通过 `XCE_SERVER_DIR`（未设置时按自身二进制路径推断）去找 `xcecd_worker`，推断结果是仓库路径 `/home/jsxing/allwork/...`。该路径的父目录 `/home/jsxing` 权限为 `750`（仅属主可进入），`admin` 无法穿越，于是 `exec` 失败 → `Permission denied` → sh 退出码 126。`namei -l` 可确认：
|
||||||
|
|
||||||
|
```
|
||||||
|
drwxr-x--- jsxing jsxing home/jsxing <- 其他用户无法进入
|
||||||
|
-rwxr-xr-x jsxing jsxing .../xcecd_worker
|
||||||
|
```
|
||||||
|
|
||||||
|
> 相关背景：`controller_connector.c` 里 bug-1507 的注释只处理了"执行用户不存在"（systemd 退出码 217 EXIT_USER），未覆盖"二进制路径不可达"（126）这一种情况。
|
||||||
|
|
||||||
|
**修复**（`all.sh`）：
|
||||||
|
|
||||||
|
1. 新增 customize 变量 `g_worker_bin_dir="/opt/xce/bin"`（所有作业用户都能到达的目录，XCE 标准布局）。
|
||||||
|
2. `svc_prepare_worker` 用 `sudo install -m 0755` 把 `xcecd_worker` 装到 `${g_worker_bin_dir}`（源更新才重装，幂等）。
|
||||||
|
3. 新增 `svc_env_worker`：启动 worker 时注入 `XCE_SERVER_DIR=${g_worker_bin_dir}`（`svc_start` 对 sudo 服务通过 `sudo -n env ...` 注入），使 `dbus_start_worker` 使用安装好的二进制而不是仓库路径。
|
||||||
|
|
||||||
|
**验证**：`./all.sh restart worker` 后：
|
||||||
|
|
||||||
|
- `/opt/xce/bin/xcecd_worker` 存在且 `admin` 可执行（`sudo -u admin /opt/xce/bin/xcecd_worker -h` 正常）。
|
||||||
|
- `xcecd` 进程环境含 `XCE_SERVER_DIR=/opt/xce/bin`。
|
||||||
|
- `xcecli jobs submit sleep 60` 状态保持 `RUN` 直到跑满 60s，controller 日志 `on_job_finish ... exit code 0` 干净结束。
|
||||||
|
|
||||||
|
> 说明：worker 日志目录不是问题——`xcecd` 启动时会 `chmod(logdir, 0777)` 并预创建 `xcecd_worker.log`（0666），执行用户可写。
|
||||||
|
|
||||||
|
### 9.5 作业无法调度
|
||||||
|
修改 src/configs/*.yaml 后必须重新 ./all.sh config deploy —— cluster.bin 是编译产物，不会自动跟随源文件。
|
||||||
|
|
||||||
|
## 10. 使用注意事项
|
||||||
|
|
||||||
|
1. **sudo**：worker 需要 root。要求 `sudo -n` 可用（NOPASSWD 或缓存凭证）；长期使用建议配置 `/etc/sudoers` 的 NOPASSWD。
|
||||||
|
2. **构建环境**：构建需 `nix develop` 环境（提供 meson、openssl 等工具）。
|
||||||
|
3. **日志查看**：服务启动失败时，`status` 会显示日志最后一行，完整日志在 `logs/<服务名>.log`。
|
||||||
|
4. **PID 文件**：`run/<服务名>.pid`；stale pid 文件会被自动清理。
|
||||||
|
5. **测试服务管理**：如需测试 start/stop 机制，建议用临时副本 + 无害二进制（如 `/bin/sleep` + 自定 `svc_args_<测试名>`），不要污染真实服务。
|
||||||
|
|
||||||
|
## 11. 目录结构速查
|
||||||
|
|
||||||
|
```
|
||||||
|
all.sh # 主脚本
|
||||||
|
all_README.md # 本文档
|
||||||
|
src/build/ # 开发版构建产物
|
||||||
|
controller/xce_controller
|
||||||
|
controller/xce_test
|
||||||
|
scheduler/xce_scheduler
|
||||||
|
worker/xcecd
|
||||||
|
frontend/apiserver/xce_apiserver
|
||||||
|
frontend/xcecli/xcecli
|
||||||
|
src/build-prod/ # 生产版构建产物
|
||||||
|
deploy/ # 统一部署根目录（${g_deploy_dir}）
|
||||||
|
conf/ # 生成配置
|
||||||
|
worker.yaml # worker 配置（svc_prepare_worker 生成，log_dir/log_level 跟随 all.sh）
|
||||||
|
run/ # PID 文件 / 证书 / 数据
|
||||||
|
controller.pid # PID 文件
|
||||||
|
scheduler.pid
|
||||||
|
worker.pid
|
||||||
|
apiserver.pid
|
||||||
|
certs/ # TLS 证书（自动生成）
|
||||||
|
controller/ # controller 数据目录
|
||||||
|
ctrl-1/conf/ # cluster.bin.* + current（config deploy 生成）
|
||||||
|
ctrl-1/data/ # WAL / snap / ha-term
|
||||||
|
apiserver/ # apiserver top 目录
|
||||||
|
config/ # app.yaml / logger.yaml / rbac.yaml / auth.yaml（自动生成）
|
||||||
|
logs/ # apiserver 自身日志
|
||||||
|
logs/ # 服务日志
|
||||||
|
controller.log
|
||||||
|
scheduler.log
|
||||||
|
worker.log
|
||||||
|
apiserver.log
|
||||||
|
~/.xce/config.json # xcecli 登录令牌（JWT，login 后生成）
|
||||||
|
```
|
||||||
|
|
||||||
|
> 旧路径 `/opt/xce/conf/worker.yaml`（需 root 写入）已不再被 all.sh 引用；worker 配置统一在 `deploy/conf/worker.yaml`。
|
||||||
|
|
||||||
|
> 作业 worker 二进制例外：`xcecd_worker` 被 systemd 以作业执行用户身份运行，必须装在所有人可访问的位置（默认 `/opt/xce/bin/xcecd_worker`，见 `g_worker_bin_dir` 与 9.4）。
|
||||||
|
|
||||||
|
# xcecli cases
|
||||||
|
## jobs
|
||||||
|
```bash
|
||||||
|
xcecli jobs submit -J 'A[1-10]' sleep 222
|
||||||
|
xcecli jobs list
|
||||||
|
xcecli hosts list
|
||||||
|
xcecli jobs submit sleep 100
|
||||||
|
```
|
||||||
1
all_note.md
Normal file
1
all_note.md
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
all.sh config deploy生成的配置文件worker.yaml位置在/opt/xce/conf/worker.yaml，这个不太好，我想把配置都生成到当前all.sh的同一个目录的某个目录下面。所有的配置都放到这下面，还有logs和run也放这个目录。这样就不会出现我现在worker启动不了的问题。因为worker启动配置/opt/xce/conf/worker.yaml里面引用的还是之前的目录。
|
||||||
|
|
@ -54,6 +54,16 @@ hosts:
|
||||||
model: "x86_64-linux-gnu"
|
model: "x86_64-linux-gnu"
|
||||||
max_slot: 8
|
max_slot: 8
|
||||||
|
|
||||||
|
# test01: the local single-node worker (xcecd reports gethostname()=test01).
|
||||||
|
# Without an explicit hostconfig the worker falls back to "default"
|
||||||
|
# (max_slot: -1 -> auto-detect from worker attrs), which has left slots
|
||||||
|
# unset (resvals empty) so the scheduler sees slots total=0 and every job
|
||||||
|
# stays PEND (bug-1494). Give it a concrete max_slot like t1/t2/t3.
|
||||||
|
- hostname: test01
|
||||||
|
type: "x86_64-linux-gnu"
|
||||||
|
model: "x86_64-linux-gnu"
|
||||||
|
max_slot: 8
|
||||||
|
|
||||||
|
|
||||||
hostgroups:
|
hostgroups:
|
||||||
- groupname: hg1
|
- groupname: hg1
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue