forked from lei.chen/myxce
docs(scheduler-agent): add cluster deployment + chatbot verification manual
New src/docs/chatbot-verification.md: reproducible steps to deploy the scheduler-agent stack on the t1/t2/t3 test cluster (AppImage build, store sync, controller/apiserver/scheduler/worker startup) and verify Phase 0-1 (login, mcp_operator API key, read ops, submit). Documents the vcpus quirk, setsid launch pattern, conf/current pointer mechanism, and known bugs (job-control HTTP 500 = bug-1490, HA election deadlock). Also add the manual to the design doc's related-documents index. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
lei.chen
parent
9fe14c1512
commit
56da673b22
2 changed files with 277 additions and 0 deletions
|
|
@ -584,6 +584,7 @@ Agent loop 跨多轮调度器轮次保持上下文：
|
|||
| 文档 | 位置 | 说明 |
|
||||
|------|------|------|
|
||||
| 本文档 | `doc/Scheduler-Agent-设计文档.md` | 设计权威来源 |
|
||||
| 集群部署与验证手册 | `src/docs/chatbot-verification.md` | t1/t2/t3 集群启动 + Phase 0-1 验证步骤（2026-08-01） |
|
||||
| CLAUDE.md（根目录） | `CLAUDE.md` | AI 实施向导（含最新进度） |
|
||||
| 架构图源文件 | `architecture_overview` | Graphviz DOT |
|
||||
| 架构图渲染 | `architecture_overview.svg` | 可视化架构图 |
|
||||
|
|
|
|||
276
src/docs/chatbot-verification.md
Normal file
276
src/docs/chatbot-verification.md
Normal file
|
|
@ -0,0 +1,276 @@
|
|||
# Chatbot (Scheduler-Agent) 集群部署与验证手册
|
||||
|
||||
> 版本：2026-08-01
|
||||
> 分支：`chatbot`（myxce fork）
|
||||
> 目标集群：t1 (172.31.1.30) / t2 (172.31.1.31) / t3 (172.31.1.32)
|
||||
|
||||
本文档记录 scheduler-agent（chatbot 分支）在真实测试集群上的**部署**与**验证**步骤，可完整复现本次会话的操作。验证覆盖 Phase 0（只读）+ Phase 1（写操作：submit），以及发现的 apiserver 预存 bug。
|
||||
|
||||
---
|
||||
|
||||
## 1. 架构总览
|
||||
|
||||
```
|
||||
用户自然语言 → Claude Code → MCP stdio → mcp-server → schedulerclient → apiserver REST API → controller
|
||||
（读：6 只读工具 + 写：4 写工具）
|
||||
```
|
||||
|
||||
| 组件 | 二进制 | 部署位置 | 端口 |
|
||||
|------|--------|---------|------|
|
||||
| controller | `xce_controller` | t1（单节点，非 HA） | 9990 |
|
||||
| apiserver | `xce_apiserver` | t1 | 8080 |
|
||||
| scheduler | `xce_scheduler` | t1 | 7890 |
|
||||
| worker | `xcecd` | t1 / t2 / t3 | — |
|
||||
|
||||
**拓扑说明**：本次验证采用 **t1 单节点 controller**（3 节点 HA 选举因遗留 term 状态卡在 STANDBY，见 §6 已知问题）。单节点用 `--peers` 省略即可。
|
||||
|
||||
---
|
||||
|
||||
## 2. 前置：构建 AppImage
|
||||
|
||||
AppImage 是部署的载体——`/opt/newxce/` 目录 = AppImage 解包后的 store 树 + `run.sh` 调度器。
|
||||
|
||||
```bash
|
||||
# 在本地仓库根目录
|
||||
cd /home/lchen/code/xce/src
|
||||
|
||||
# 1. 构建 xce wrapper（含 xcecore 闭包）
|
||||
nix build .#xce
|
||||
|
||||
# 2. 打包 AppImage（280MB，mksquashfs 单遍）
|
||||
# 输入：新 xce 闭包（nix path-info -r）+ AppRun + entrypoint
|
||||
# 详见 /home/lchen/xce-appimage/package.sh
|
||||
./package.sh # → /home/lchen/xce-appimage/xce-x86_64.AppImage
|
||||
```
|
||||
|
||||
**AppImage 结构**（与旧版一致，验证 `unsquashfs -offset 1002016 -l`）：
|
||||
```
|
||||
squashfs-root/AppRun
|
||||
squashfs-root/entrypoint # → 指向 xcecore 的 bin/xce
|
||||
squashfs-root/mountroot
|
||||
squashfs-root/nix/store/… # 99 个 store 路径闭包
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 部署到 t1/t2/t3
|
||||
|
||||
### 3.1 部署 AppImage 结构
|
||||
|
||||
三台主机均为 `lchen` 用户，SSH 免密（`~/.ssh/id_rsa`），有 passwordless sudo。
|
||||
|
||||
```bash
|
||||
# t1 完整部署（store 树 + run.sh + config）
|
||||
# /opt/newxce/ 结构：
|
||||
# bin/xce # AppImage 二进制（280MB 静态 ELF）
|
||||
# app/nix/store/… # 99 个 store 路径（xcecore + 依赖）
|
||||
# run.sh # daemon 调度器（controller/worker/scheduler/apiserver）
|
||||
# config/ # 集群 + apiserver 配置
|
||||
|
||||
# t2/t3 从 t1 同步 store 树（worker 需要 xcecd + 依赖闭包）
|
||||
ssh t1 'rsync -a /opt/newxce/app/ t2:/opt/newxce/app/ && rsync -a /opt/newxce/run.sh t2:/opt/newxce/run.sh'
|
||||
ssh t1 'rsync -a /opt/newxce/app/ t3:/opt/newxce/app/ && rsync -a /opt/newxce/run.sh t3:/opt/newxce/run.sh'
|
||||
```
|
||||
|
||||
### 3.2 停旧 systemd 服务（可选，若存在旧 XCE）
|
||||
|
||||
```bash
|
||||
ssh t1 'sudo systemctl stop scheduler.service workerd.service; sudo systemctl disable scheduler.service workerd.service'
|
||||
ssh t2 'sudo systemctl stop workerd.service; sudo systemctl disable workerd.service'
|
||||
```
|
||||
|
||||
### 3.3 准备 controller workdir + 证书 + cluster.bin
|
||||
|
||||
```bash
|
||||
# 每台 controller 需要 workdir + TLS 证书（HA 复制用；单节点也生成以防切换）
|
||||
for h in t1 t2 t3; do
|
||||
ssh $h "sudo mkdir -p /opt/xce/controller/conf /opt/xce/controller/data /opt/xce/controller/certs /opt/xce/logs \
|
||||
&& sudo chown -R lchen:lchen /opt/xce/controller /opt/xce/logs \
|
||||
&& openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
|
||||
-keyout /opt/xce/controller/certs/xce.key -out /opt/xce/controller/certs/xce.crt -subj '/CN=xce'"
|
||||
done
|
||||
|
||||
# 生成 cluster.bin（t1 上，用 AppImage 内的 xceadmin）
|
||||
ssh t1 '/opt/newxce/app/nix/store/<xcecore-hash>/bin/xceadmin config build /opt/newxce/config -o /tmp/cluster.bin'
|
||||
|
||||
# 部署到 t1 的 controller conf（指针文件 + 版本化 bin）
|
||||
# tag = cluster.bin 的 sha256 前 12 位（上面命令 stderr 输出）
|
||||
ssh t1 'sudo mkdir -p /opt/xce/controller/t1/conf \
|
||||
&& sudo cp /tmp/cluster.bin /opt/xce/controller/t1/conf/cluster.bin.<tag> \
|
||||
&& printf "cluster.bin.<tag>\n" | sudo tee /opt/xce/controller/t1/conf/current'
|
||||
```
|
||||
|
||||
> **conf/current 指针机制**：`<workdir>/<myname>/conf/current` 内容为 `cluster.bin.<12hex>`，实际配置在 `cluster.bin.<12hex>`。controller 通过 `load_primary_config` 读取。tag 是配置 SHA-256 前 12 位。
|
||||
|
||||
### 3.4 准备 apiserver 配置
|
||||
|
||||
```bash
|
||||
# apiserver 需要 <topdir>/config/{app,auth,rbac,logger}.yaml
|
||||
ssh t1 'sudo mkdir -p /opt/xce/apiserver/config \
|
||||
&& cp /opt/newxce/config/app.yaml /opt/newxce/config/auth.yaml \
|
||||
/opt/newxce/config/rbac.yaml /opt/newxce/config/logger.yaml \
|
||||
/opt/xce/apiserver/config/'
|
||||
|
||||
# 注意：/opt/newxce/config/rbac.yaml 是旧版（无 mcp_operator）。
|
||||
# 必须用本地仓库的 rbac.yaml（含 mcp_operator + queue:control）
|
||||
scp src/frontend/apiserver/config/rbac.yaml t1:/opt/xce/apiserver/config/rbac.yaml
|
||||
```
|
||||
|
||||
### 3.5 启动 daemon
|
||||
|
||||
所有 daemon 通过 `/opt/newxce/run.sh` 启动（设置 LD_LIBRARY_PATH + 正确的 glibc 加载器）。用 `setsid … &` 完全脱离 SSH 会话。
|
||||
|
||||
```bash
|
||||
# t1: controller（单节点，省略 --peers = 非 HA）
|
||||
ssh t1 'sudo bash -c "cd /opt/newxce && setsid ./run.sh controller -n t1 \
|
||||
-w /opt/xce/controller --certdir /opt/xce/controller/certs \
|
||||
-l /opt/xce/logs/xce_controller.log --vcpus 4 -v </dev/null >/dev/null 2>&1 &"'
|
||||
|
||||
# t1: apiserver（backend 指向 localhost:9990，配置在 /opt/xce/apiserver）
|
||||
ssh t1 'sudo bash -c "cd /opt/newxce && setsid ./run.sh apiserver -d /opt/xce/apiserver </dev/null >/dev/null 2>&1 &"'
|
||||
|
||||
# t1: scheduler + worker
|
||||
ssh t1 'sudo bash -c "cd /opt/newxce && setsid ./run.sh scheduler --peers t1=127.0.0.1:9990 -l /opt/xce/logs/xce_scheduler.log -v </dev/null >/dev/null 2>&1 &"'
|
||||
ssh t1 'sudo bash -c "cd /opt/newxce && setsid ./run.sh worker -c /opt/xce/config/worker.yaml </dev/null >/dev/null 2>&1 &"'
|
||||
|
||||
# t2/t3: worker
|
||||
ssh t2 'sudo bash -c "cd /opt/newxce && setsid ./run.sh worker -c /opt/xce/config/worker.yaml </dev/null >/dev/null 2>&1 &"'
|
||||
ssh t3 'sudo bash -c "cd /opt/newxce && setsid ./run.sh worker -c /opt/xce/config/worker.yaml </dev/null >/dev/null 2>&1 &"'
|
||||
```
|
||||
|
||||
> **启动要点**：
|
||||
> - 必须加 `--vcpus 4`（`--vcpus` 缺失会导致 vtask pool 未初始化，`vtask unavailable (pool=nil)`，无法完成选举）
|
||||
> - controller 以 root 运行（`sudo`）
|
||||
> - `setsid … &` 使进程脱离会话，否则 SSH 会话会挂起
|
||||
> - 检查进程用 `sudo fuser 9990/tcp` / `sudo ps aux | grep ld-linux.*xce`（进程名是 `ld-linux-x86-64.so.2`，不是 `xce_controller`）
|
||||
|
||||
### 3.6 启动验证
|
||||
|
||||
```bash
|
||||
# 端口监听
|
||||
ssh t1 'sudo ss -tlnp | grep -E ":9990|:8080|:7890"'
|
||||
|
||||
# controller 应为 CTRL_PRIMARY
|
||||
ssh t1 'sudo grep -E "CTRL_PRIMARY|become_leader" /opt/xce/logs/xce_controller.log | tail'
|
||||
|
||||
# apiserver 响应
|
||||
ssh t1 'curl -s http://localhost:8080/xce/v1/cluster' # 应返回 401（需认证）而非连接失败
|
||||
```
|
||||
|
||||
预期 controller 日志：
|
||||
```
|
||||
raft_cb_become_leader: → CTRL_PRIMARY (term=1, first_boot=1)
|
||||
committed_lsn 0 -> 1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 验证 Chatbot（Phase 0-1）
|
||||
|
||||
### 4.1 登录 + 生成 mcp_operator API Key
|
||||
|
||||
```bash
|
||||
# 1. 登录（auth.yaml 有 admin/admin, cluster_admin）
|
||||
TOKEN=$(curl -s -X POST http://localhost:8080/xce/v1/auth/logon \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"username":"admin","password":"admin"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['token'])")
|
||||
|
||||
# 2. 生成 mcp_operator key（服务账号，供 mcp-server 用）
|
||||
curl -s -X POST http://localhost:8080/xce/v1/apikeys \
|
||||
-H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" \
|
||||
-d '{"username":"scheduler-agent","role_name":"mcp_operator","description":"phase1","expiry_days":365}'
|
||||
# → 返回 xce_ak_<64hex>
|
||||
```
|
||||
|
||||
### 4.2 Phase 0 — 只读验证
|
||||
|
||||
```bash
|
||||
KEY="xce_ak_<生成的key>"
|
||||
BASE="http://localhost:8080"
|
||||
|
||||
# 全部应返回 200
|
||||
curl -s -H "Authorization: $KEY" "$BASE/xce/v1/queues"
|
||||
curl -s -H "Authorization: $KEY" "$BASE/xce/v1/hosts"
|
||||
curl -s -H "Authorization: $KEY" "$BASE/xce/v1/jobs"
|
||||
curl -s -H "Authorization: $KEY" "$BASE/xce/v1/resources/definition"
|
||||
```
|
||||
|
||||
### 4.3 Phase 1 — 写操作验证
|
||||
|
||||
```bash
|
||||
# submit → 201，返回 jobid
|
||||
JOB=$(curl -s -X POST -H "Authorization: $KEY" -H "Content-Type: application/json" \
|
||||
-d '{"command":"sleep 60","job_name":"verify","queue_name":"normal"}' \
|
||||
"$BASE/xce/v1/jobs/create" | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['jobid'])")
|
||||
|
||||
# 作业应进入 controller（PEND）
|
||||
curl -s -H "Authorization: $KEY" "$BASE/xce/v1/jobs"
|
||||
```
|
||||
|
||||
**当前限制**：`hold/release/cancel`（job control）返回 **HTTP 500** —— apiserver 预存 bug（bug-1490），见 §6。
|
||||
|
||||
---
|
||||
|
||||
## 5. schedulerclient / mcp-server 本地验证
|
||||
|
||||
在本地仓库（`chatbot` 分支）可独立运行 Go 测试（不需要集群）：
|
||||
|
||||
```bash
|
||||
# schedulerclient（读 + 写）
|
||||
cd src/frontend/schedulerclient && go vet ./... && go build ./... && go test ./... -count=1
|
||||
|
||||
# mcp-server（含模板 + handlers）
|
||||
cd src/frontend/mcp-server && go vet ./... && go build ./... && go test ./... -count=1
|
||||
|
||||
# apiserver RBAC
|
||||
cd src/frontend/apiserver && CGO_ENABLED=0 go test ./controllers/... -run 'RBAC|APIKey' -count=1
|
||||
```
|
||||
|
||||
schedulerclient 用 API key 连真实集群（若集群在跑）：
|
||||
```bash
|
||||
cd src/frontend/schedulerclient
|
||||
cat > /tmp/demo.go <<'EOF'
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
sclient "schedulerclient"
|
||||
)
|
||||
|
||||
func main() {
|
||||
c := sclient.NewClient("http://172.31.1.30:8080", "xce_ak_<key>")
|
||||
qs, _ := c.ListQueues(context.Background())
|
||||
for _, q := range qs {
|
||||
fmt.Printf("queue %s status=%s run=%d pend=%d\n", q.Name, q.Status, q.RunningJobs, q.PendingJobs)
|
||||
}
|
||||
}
|
||||
EOF
|
||||
go run /tmp/demo.go
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 已知问题
|
||||
|
||||
| 问题 | 状态 | Bug |
|
||||
|------|------|-----|
|
||||
| **job control (cancel/hold/release) HTTP 500** | 未修。请求未达 controller（controller 日志无 control 事件）。已修 `job_status` int64 断言（从 "Invalid job_status" 变为 500），500 本体待排查 | bug-1490 |
|
||||
| **3 节点 HA 选举死锁** | 遗留 term 状态（`ha-term.bin` 残留 `voted_for=t1, term=1`）+ standby 无 pool 导致所有节点 STANDBY。单节点绕过 | 记录在会话 |
|
||||
| **apikeys 路由 whitelist** | 已修：`WhitelistPOSTURL` 加 `/xce/v1/apikeys` | bug-1489（已修） |
|
||||
| **mcp_operator 缺 queue:control** | 已修：rbac.yaml 加 `queue:control`（job-control handler 也检查它） | — |
|
||||
| **HA 3 节点需 xceadmin push** | `xceadmin config push` 有 discovery.py 解包 bug（deployment.md 已知） | 已知 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 快速参考
|
||||
|
||||
| 操作 | 命令 |
|
||||
|------|------|
|
||||
| 查 controller 进程 | `sudo fuser 9990/tcp` |
|
||||
| 查 apiserver | `curl -s http://localhost:8080/xce/v1/cluster` |
|
||||
| controller 日志 | `sudo tail /opt/xce/logs/xce_controller.log` |
|
||||
| apiserver 日志 | `sudo tail /opt/xce/apiserver/logs/apiserver_out.log` |
|
||||
| 停 controller | `sudo fuser -k 9990/tcp` |
|
||||
| 重启 apiserver | `kill $(fuser 8080/tcp); cd /opt/newxce && setsid ./run.sh apiserver -d /opt/xce/apiserver &` |
|
||||
| 本地 Go 测试 | 见 §5 |
|
||||
Loading…
Add table
Add a link
Reference in a new issue