docs(advisor): manual demo case handbook for t1/t2/t3 cluster
Four demo scenarios, all live-verified on the test cluster (2026-10-19/20): 1. pend_stall diagnosis (most reliable; includes the change-triggered second-fire follow-up demo) 2. real-LLM cross-field inconsistency catch (glm queue-accounting finding, reproduced twice) 3. worker_health node-down detection (fastest; documents the repeat-outage dedup gap found while verifying: lastKey is never cleared on recovery, so a second identical outage stays silent — one-line fix candidate) 4. MCP scheduler_advisor_findings query (zero-risk closer, auditability) Plus pre-flight checks, advisor start command, known pitfalls table (in-memory API keys die on apiserver restart, glm timeouts degrade to echo, SSE listener timeout vs 5-min stall threshold), and cleanup checklist. Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen
parent
1691e49a05
commit
321544f7be
1 changed files with 190 additions and 0 deletions
190
src/docs/advisor-demo-cases.md
Normal file
190
src/docs/advisor-demo-cases.md
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
# Scheduler-Advisor 手工演示用例手册
|
||||
|
||||
备选演示场景，全部在 t1/t2/t3 测试集群实测通过（2026-10-19/20，分支
|
||||
feature/scheduler-advisor，advisor 提交 1691e49a）。按演示时长和戏剧效果排
|
||||
序：用例 1 最稳（确定性回显、零外部依赖），用例 2 最出彩（真 LLM 抓跨字段
|
||||
不一致），用例 3 最直观（节点故障）。
|
||||
|
||||
## 前置条件（演示前检查一次）
|
||||
|
||||
```bash
|
||||
# 1. 六个 daemon 全活：apiserver / controller / scheduler / xcecd(t1,t2,t3)
|
||||
ps aux | grep -E "[x]ce_apiserver|[x]ce_controller|[x]ce_scheduler" | wc -l # ≥3
|
||||
ssh t2 'ps aux | grep [x]cecd | wc -l' # ≥1（t3 同理；t1 本机直接看）
|
||||
|
||||
# 2. 三台主机 OK
|
||||
KEY=$(cat /tmp/advisor_apikey_v2)
|
||||
curl -s -H "Authorization: Bearer $KEY" http://172.31.1.30:8080/xce/v1/hosts \
|
||||
| python3 -c "import sys,json;[print(r['Data']['host_name'],r['Data']['host_status']) for r in json.load(sys.stdin)['data']['rows']]"
|
||||
# 期望：t1 OK / t2 OK / t3 OK
|
||||
|
||||
# 3. advisor 存活且在扫描（60s 一轮）
|
||||
ps aux | grep [x]ce-advisor | wc -l # ≥1
|
||||
tail -2 /opt/xce/logs/advisor.log # 期望 "scan clean (0 pend, 0 run, 3 nodes)"
|
||||
|
||||
# 4. LLM 路由可用（用例 2 需要；不通则 advisor 自动降级 echo，演示仍可做但亮点变少）
|
||||
curl -s -m 10 http://127.0.0.1:6789/v1/models -H "Authorization: Bearer sk-litellm-router" | head -c 120
|
||||
|
||||
# 5. API key 是否有效（apiserver 重启过就失效，见"坑"一节）
|
||||
curl -s -H "Authorization: Bearer $KEY" "http://172.31.1.30:8080/xce/v1/jobs?limit=1" | head -c 80
|
||||
```
|
||||
|
||||
advisor 启动命令（若未运行）：
|
||||
|
||||
```bash
|
||||
env XCE_API_URL=http://172.31.1.30:8080 \
|
||||
XCE_API_KEY=$(cat /tmp/advisor_apikey_v2) \
|
||||
ADVISOR_LLM_BASE_URL=http://127.0.0.1:6789/v1 \
|
||||
ADVISOR_LLM_API_KEY=sk-litellm-router \
|
||||
ADVISOR_LLM_MODEL=glm-5-3-flash \
|
||||
ADVISOR_INTERVAL=60s \
|
||||
ADVISOR_LOG=/opt/xce/logs/advisor-decisions.jsonl \
|
||||
nohup setsid /tmp/xce-advisor-new run >> /opt/xce/logs/advisor.log 2>&1 &
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 用例 1：PEND 作业卡死诊断（最稳，5-8 分钟）
|
||||
|
||||
**故事**：一个作业因为资源需求永远无法满足而卡在队列里，无人察觉。advisor
|
||||
自动发现、诊断、给出建议。
|
||||
|
||||
**步骤**：
|
||||
|
||||
```bash
|
||||
# 1. 提交一个"不可能满足"的作业（r1m>9999 可解析但无节点能满足）
|
||||
export PATH=/opt/xce/bin:$PATH
|
||||
xce login -u admin -p admin --url http://172.31.1.30:8080
|
||||
xce jobs submit -J demo-pend -q normal -R "select[r1m > 9999]" "sleep 300"
|
||||
# 输出：Job <...> is submitted to default queue <normal>
|
||||
|
||||
# 2. （可选但推荐）开一个 SSE 监听窗，现场看到 advisor.finding 事件飞出来
|
||||
KEY=$(cat /tmp/advisor_apikey_v2)
|
||||
curl -s -N -H "Authorization: Bearer $KEY" http://172.31.1.30:8080/xce/v1/events
|
||||
|
||||
# 3. 等待：提交后 5 分钟（stall 阈值）+ 最多 1 分钟（扫描间隔）
|
||||
# 即最迟 ~6 分钟后看到事件和决策
|
||||
```
|
||||
|
||||
**预期输出**（advisor 日志 `/opt/xce/logs/advisor.log`）：
|
||||
|
||||
```
|
||||
advisor: pend_stall [pending=1 stalled=1] -> 1 finding(s) (model=glm-5-3-flash, ~20s, degraded=false)
|
||||
```
|
||||
|
||||
SSE 窗口出现 `data: {"type":"advisor.finding","job_status":"pend_stall",...}`；
|
||||
MCP 工具 `scheduler_advisor_findings` 同时能查到同一条记录（见用例 4）。
|
||||
|
||||
**卖点**：从作业卡死到诊断推送 ≤6 分钟，全程无人操作、只读、零侵入。
|
||||
|
||||
**去重语义现场演示（加分项）**：什么都不做，advisor 不会重复刷屏（同一情形
|
||||
永不重报）；此时**再提交第二个**卡死作业（`-n 64`）：
|
||||
|
||||
```bash
|
||||
xce jobs submit -J demo-pend-2 -q normal -n 64 "sleep 300"
|
||||
```
|
||||
|
||||
下一个扫描轮（≤1 分钟）立即出**新**决策（变化即触发）——演示"恶化会被
|
||||
即时上报，而不是被冷却时间捂住"。
|
||||
|
||||
**清理**：
|
||||
|
||||
```bash
|
||||
xce jobs list # 拿到 JOBID
|
||||
xce jobs kill <JOBID1> [JOBID2]
|
||||
# 等 1-2 分钟，advisor 日志回到 "scan clean"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 用例 2：真 LLM 抓跨字段不一致（最出彩，需 LLM 路由）
|
||||
|
||||
与用例 1 同一操作，但重点展示 LLM 的增值：glm-5-3-flash 在同一份快照上
|
||||
输出了 echo 规则给不出的**第二条** finding——"cluster 报 2 个 pending，
|
||||
但队列 normal 报 0 pending 0 running，记账不一致"。这是真实观测（两次
|
||||
独立运行都出现），evidence 全部逐字可验、unverified=false。
|
||||
|
||||
**讲法**：echo 是基线（兜底、零成本），LLM 是增量（跨字段关联）；防火墙
|
||||
保证 LLM 说的话每个标识符都能在快照里找到原文，编造就降级标记。
|
||||
|
||||
**风险预案**：glm 路由偶发超时（30s 超时后自动降级 echo，决策标记
|
||||
degraded=true）——现场反而是卖点："外部模型挂了，advisor 没挂"。
|
||||
|
||||
---
|
||||
|
||||
## 用例 3：节点故障检测（最快出活，2-3 分钟）
|
||||
|
||||
**故事**：一台节点挂了/被维护关了，advisor 立即感知容量损失。
|
||||
|
||||
```bash
|
||||
# 1. 关闭 t3（模拟节点故障）
|
||||
export PATH=/opt/xce/bin:$PATH
|
||||
xce hosts close t3
|
||||
# xce hosts list 显示 t3 CLOSED_BY_ADMIN
|
||||
|
||||
# 2. 等 ≤1 个扫描周期（60s），advisor 日志：
|
||||
# advisor: worker_health [t3:down] -> 1 finding(s) (model=glm-5-3-flash, ~12s, degraded=false)
|
||||
# SSE advisor.finding：
|
||||
# "Node t3 is down while t1 and t2 are idle; cluster capacity is
|
||||
# reduced to 2 of 3 hosts..."
|
||||
# 建议语提示确认是否计划内维护。
|
||||
|
||||
# 3. 恢复
|
||||
xce hosts open t3
|
||||
```
|
||||
|
||||
**实测坑（演示前必读）**：**重复故障会被去重吞掉**。当前实现的 dedup key
|
||||
在节点恢复（全健康）时**不清除**——同一节点第二次 close 不会再报（实测
|
||||
01:20 复测确认）。两个选择：
|
||||
- 演示只做**一次** close（安全）；
|
||||
- 或者先修这个缺口再演示：`fire()` 之后任何一轮全健康时应把该场景的
|
||||
lastKey 清掉（一行改动 + 测试），修复后"每次新故障都报"。
|
||||
|
||||
**清理**：`xce hosts open t3`，等日志回 scan clean。
|
||||
|
||||
---
|
||||
|
||||
## 用例 4：MCP 工具查询（收尾，1 分钟，零风险）
|
||||
|
||||
不需要制造任何异常，直接展示决策日志可查（适合给领导看"可审计"）：
|
||||
|
||||
```bash
|
||||
cd /tmp && env XCE_API_URL=http://172.31.1.30:8080 \
|
||||
XCE_API_KEY=$(cat /tmp/advisor_apikey_v2) \
|
||||
ADVISOR_LOG=/opt/xce/logs/advisor-decisions.jsonl \
|
||||
/tmp/xce-mcp-server-new <<'EOF'
|
||||
{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"demo","version":"1.0"}}}
|
||||
{"jsonrpc":"2.0","id":2,"method":"notifications/initialized"}
|
||||
{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"scheduler_advisor_findings","arguments":{"tail":5}}}
|
||||
EOF
|
||||
```
|
||||
|
||||
在 Claude Code 里接上 mcp-server 后就是自然语言："advisor 最近发现了什么？"
|
||||
每条记录含 ts/scenario/provider/model/延迟/证据，与决策日志一一对应——
|
||||
"当时它看到了什么、说了什么"可完整回放。
|
||||
|
||||
---
|
||||
|
||||
## 已知坑与预案（演示前一天过一遍）
|
||||
|
||||
| 坑 | 影响 | 预案 |
|
||||
|---|---|---|
|
||||
| **apiserver 重启使所有 API key 失效**（key 是纯内存 map，无持久化） | 所有 curl/xce 401 | 重新发：logon 拿 admin token → `POST /xce/v1/apikeys`（role_name=advisor_agent，username=advisor-svc）→ 存回 `/tmp/advisor_apikey_v2` → **重启 advisor** |
|
||||
| glm 路由超时（偶发） | 用例 2 降级为 echo（degraded=true） | 顺势讲兜底设计；或演示前 `curl /v1/models` 预检 |
|
||||
| SSE 监听用 `curl -m 180` 会在 5min 阈值前断 | 以为推送没到 | 监听窗口用 `-m 600` 以上，或用 MCP 工具事后查 |
|
||||
| stall 阈值 5 分钟 | 用例 1 有 5-6 分钟干等 | 先讲架构图/MCP 工具填这段时间；或把演示作业提前 5 分钟提交 |
|
||||
| 重复节点故障被 dedup 吞 | 用例 3 二连击失败 | 只关一次；或先修 lastKey 清除缺口 |
|
||||
| advisor 进程死了 | 一切不输出 | 用前置检查第 3 步确认；重启命令见上 |
|
||||
|
||||
## 演示后清场清单
|
||||
|
||||
```bash
|
||||
# 1. kill 所有演示作业（确认 EXIT）
|
||||
export PATH=/opt/xce/bin:$PATH && xce jobs list
|
||||
# 2. 确认 t1/t2/t3 全 OK
|
||||
xce hosts list
|
||||
# 3. advisor 日志回到 "scan clean (0 pend, 0 run, 3 nodes)"
|
||||
tail -2 /opt/xce/logs/advisor.log
|
||||
# 4. （可选）归档本次演示的决策记录
|
||||
cp /opt/xce/logs/advisor-decisions.jsonl /tmp/demo-decisions-$(date +%Y%m%d-%H%M).jsonl
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue