docs: update status — Phase 2 done, human jobid, priorities (2026-08-13)

- left_scheduler_work.md: Phase 2 implemented (SSE+WatchJobs+4 tools), add
  job_id_human fix + ListJobs filter fix + .mcp.json env-var note; refresh
  priority list (bug-1490/1494/Phase2 all done).
- Scheduler-Agent-设计文档.md: v1.3, status header reflects Phase 0-2 done +
  verified.
- cerebrum: user preference (apiserver owns data correctness, MCP is a thin
  reader); jobid dual-encoding lesson (reuse backend.FormatJobID).
- memory: session log.

Co-Authored-By: Claude <noreply@anthropic.com>

.wolf/cerebrum.md

@@ -11,6 +11,8 @@
 - Prefer parallel agent runs for independent analysis work; sequential for dependent tasks
 - Keep test logs for analysis, don't skip intermediate output even if a test hangs
 - NEVER push code to Rongsong's branch (`origin/working`, rshen/xce.git) unless the user explicitly allows it. We only sync Rongsong's code INTO our own branch (local working / myxce fork). When "push" is mentioned without a target repo, default to myxce (user's fork) and confirm first; `git push origin working` requires explicit permission (2026-08-01).
+- MCP server / client 是薄读取层：只读取透传数据，**apiserver 负责数据正确性**（格式、转换、权限判定都在 apiserver 侧做，不要在 mcp-server/schedulerclient 里造格式逻辑）。用户明确表达过（2026-08-13，jobid 格式问题）。
+- 提交凭证类内容（API key 等）进 git 前先三思：用环境变量引用（`.mcp.json` 用 `${XCE_API_KEY}`）而非硬编码；实时 key 被 classifier 拦截过（2026-08-13）。
 
 ## Key Learnings
 - (2026-07-28, bug-1479 ROOT-CAUSED not fixed) A permanent quorum-loss family member unrelated to admission control: when repl.c's `repl_broadcast` (live single-entry WAL tail-follow) gets an `ae_reject_logmismatch` NACK back from a standby (a genuinely NORMAL Raft outcome after rapid leader-election churn leaves a conflicting uncommitted tail entry at the same LSN with a different term — confirmed byte-precise via `xce_test waldump`: ctrl-2's own lsn=13 was its own deposed term=3 noop, while the canonical/surviving log has a DIFFERENT term=4 noop at the same lsn), NOTHING in ha.c's `repl_ack_decode_and_dispatch` NACK branch (`!success`) does anything but check for a higher observed term and then literally `WARN("...success=0 — ignoring")` — no `repl_on_ack` call (the ONLY site that decrements `pending_ack_count`), no nextIndex backoff-and-retry, no catch-up trigger. Consequence: `pending_ack_count` climbs 1-per-broadcast-attempt with zero decrements until `repl_broadcast`'s own `MAX_PENDING_ACKS=16` defensive eviction fires and marks the standby `in_catch_up=1` — which correctly excludes it from `repl_active_standby_count`/quorum, but NOTHING then drives an actual catch-up stream/snapshot-install for it (the only other `repl_complete_catchup` call site, config_repl.c's ACK-driven `catchup_lane_fn`, can never engage a standby that never sends a genuine ACK in the first place). Net effect: a standby with ONE conflicting log entry from election churn (routine, expected) becomes PERMANENTLY excluded from quorum with no self-healing path — in a 3-node cluster this is one unlucky election sequence away from total write-quorum loss. **General lesson**: when a symptom looks like "resource exhaustion" (admission cap, connection pool, whatever), always verify the actual counter's INCREMENT and DECREMENT sites are symmetric across EVERY reply outcome (success AND failure/reject), not just the happy path — a counter that only decrements on success but increments unconditionally will silently drift to its ceiling under ANY sustained failure condition, which looks identical to a genuine leak from the outside (repeated timeouts / repeated rejects) but has a completely different fix (handle the reject/failure path, not "find the missing release call"). Confirmed the ORIGINAL admission-slot-leak hypothesis for this bug was WRONG by checking `commit_resolve()`/`wait_release_locked()` — those DO release correctly on every timeout, 1:1 (17 timeouts, 17 releases) — before chasing the real mechanism.
@@ -112,6 +114,7 @@
 - (2026-07-10, bug-1433, controller-invariant-audit) `peer_ctrl_classify` (ha.c ~5935) hardcodes `VTASK_LIVENESS` for EVERY inbound peer-control packet (vote replies, PreVote replies, WAL/config ACKs) — a task class documented as "never admission-gated," i.e. never expected to block. `raft_cb_become_leader` (the raft library's `.on_become_leader` callback) is invoked synchronously from THIS dispatch path whenever a real multi-node winning vote reply arrives — NOT from the separate `VTASK_LIVE_BLOCK`-dispatched election-*start* path (`ha_dispatch_election_start`, which only covers RDHUP-failover/single-node self-grant). Any blocking call added inside `raft_cb_become_leader` (or anything it calls) is reachable on a worker class the design guarantees never blocks — same bug class as bug-1198 (closed by moving `persist_checkpoint`'s snapshot-send fdatasync off VTASK_LIVENESS entirely). Before adding ANY blocking call to the promotion path, trace peer_ctrl_classify's actual dispatch, don't trust an existing nearby comment claiming "already on VTASK_LIVE_BLOCK" — one was WRONG (see bug-1433 in buglog.json for the full trace). `persist_apply_pump_run_sync` still has this residual (bounded `pread`, lower risk than `fdatasync`, not fixed this session — see buglog.json bug-1433 for why it can't simply be removed).
 - (2026-07-10, bug-1434, controller-invariant-audit) `persist_promotion_advance_committed` and `ha_standby_advance_committed_and_apply` both did an unlocked `load; if(target<=cur) return; store` on `p->committed_lsn` — safe ONLY if truly single-writer, but `raft_cb_become_leader` calls the former BEFORE setting `role = CTRL_PRIMARY` (~170 lines later), so a still-in-flight WAL catch-up entry on another thread can call the latter concurrently during that window — classic lost-update race that can regress `committed_lsn` (monotonicity violation). General lesson: when two independent call sites do "monotonic-max store to a shared atomic, no lock" and you can't prove they're strictly serialized by role/state gating (gating checked AT CALL TIME can still race with a WINDOW where the gate value itself is about to change), use a CAS retry loop (`persist_committed_lsn_advance_atomic`), not a bare load-then-store — it's safe under ANY interleaving with zero lock-ordering risk.
 - (2026-07-28, bug-1477) `cluster_init_phase2()` (cluster.c) is the SOLE place that runs `commit_engine_init()` -- which is itself the ONLY place that registers `persist_t`'s `on_durable_advance` wake hook (persist.c:796-799 says so explicitly: "starts unregistered ... the commit engine registers it"). Without that hook, `persist_apply_pump_run`/`apply_committed_entry` NEVER runs on that node, for ANY committed LSN, for its whole process lifetime -- a node can still fully participate in WAL replication/ACK (that path is independent, in ha.c) while never materializing ANY committed state. `cluster_init_phase2` short-circuits with `-EINVAL` (cluster.c:1349-1358) whenever `c->myconf->cluster_config == NULL` (config not loaded yet) -- normal during cold boot -- and its THREE raft-driven callers (`raft_cb_become_follower` ha.c:2000-2012, `raft_cb_become_leader` ha.c:1772-1783, `ha_elect_evaluate_and_signal`'s two branches ha.c:5609-5629/5699-5710) all set `ha->phase2_done=1` UNCONDITIONALLY after calling it, with NO check of its return value -- unlike `myapp.c`'s own `skip_election` fast-path caller (~line 415-425, bug-849 lineage) which correctly gates the latch on success. Worse: a low-priority node (highest `my_priority`, longest startup stagger) that finds "leader already known" on its very first check takes `goto skip_election` and NEVER runs `ha_start_election_async` at all -- so it never reaches ANY of the three raft-callback retry opportunities, ever, for its whole life, if it never later contests its own election. If its one-shot `myapp.c` attempt races ahead of config load, `cluster_init_phase2`/`commit_engine_init` never runs again, PERIOD -- a real, permanent HA-bootstrap gap, predates config-upgrade-snapshot by ~1 month (myapp.c skip_election = commit 20e59d0b 2026-06-21; ha.c raft-callback structure = commit 8f0d2d55 task 6.2), NOT introduced by it. It stayed invisible until config-upgrade-snapshot's task 5 (`persist_checkpoint_marker_triggered`) created a NEW cross-member invariant (every member independently writes a local checkpoint at a synchronized marker-boundary LSN) that a node with a never-wired apply pump silently violates -- surfacing as a PERMANENT (not transient/racy) `ae_reject_logmismatch` catch-up wedge that superficially resembles bug-1475 but is a categorically different, unrecoverable failure (the missing entry is genuinely, permanently absent -- not merely stale bookkeeping) since a WAL_CHECKPOINT entry is local-only/non-replicated by design and can never arrive over the wire to fill the gap. Diagnostic tell: grep the suspect node's log for "commit engine wired" -- if it NEVER appears (vs. multiple times on healthy peers, once per boot/restart/election), the commit engine/apply pump never wired on that node at all.
+- (2026-08-13, scheduler-agent jobid) XCE job IDs are a 64-bit number with TWO display encodings: the raw decimal (apiserver/`73930964699447296`) and the dash form `<cluster>-<apiserver>-<timestamp>-<seq>` (xce_test/xcecli joblist, `1-0-1786585485-0`). `xce_test`'s `parse_jobid` only accepts the dash form — a user copying the raw decimal from the MCP output into `jobinfo`/`jobkill` fails. The apiserver ALREADY had the correct formatter (`backend.FormatJobID`, the C `fmt_jobid` Go equivalent, in convert_pure.go) — the fix was to surface it (`job_id_human`/`human_id` in responses) rather than duplicate it in models. Lesson: before writing a bit-manipulation formatter, grep for an existing one (`grep FormatJobID`) — the apiserver mints job IDs at CreateJob, so it already owns the format. Architectural rule from the user: **apiserver owns data correctness (formats, conversions, permission enforcement); mcp-server/schedulerclient only read and pass through** — don't add format/transform logic to the client layers.
 
 ## Do-Not-Repeat
 

.wolf/memory.md

@@ -7396,3 +7396,4 @@
 
 | Time | Action | File(s) | Outcome | ~Tokens |
 |------|--------|---------|---------|--------|
+| 02:1x | Added human-readable jobid (job_id_human/human_id) to apiserver responses (backend.FormatJobID reuse); renamed SSE perm event:subscribe->event:view; updated docs (left_scheduler_work, design doc v1.3, cerebrum) | src/frontend/apiserver, src/frontend/schedulerclient, docs | verified live (submit + get_job return 1-0-... form); all tests pass; pushed | ~90k tok |

docs/Scheduler-Agent-设计文档.md

@@ -1,9 +1,9 @@
 # Unified Scheduler Agent Framework — 设计文档
 
 > **项目代号**：scheduler-agent
-> **文档版本**：v1.2
-> **最近更新**：2026-08-01
-> **文档状态**：实施中（Phase 0-1 代码完成；集群验证：读 + submit ✅，control 有 apiserver 预存 bug 待修）
+> **文档版本**：v1.3
+> **最近更新**：2026-08-13
+> **文档状态**：实施中（**Phase 0-2 代码完成并集群验证** ✅；Phase 3-4 后端适配器未开始。Phase 0-1 E2E 2026-08-12 通过；Phase 2 SSE 事件流 + 4 高级工具 2026-08-12 实施并验证；人类可读 jobid 2026-08-13 落地）
 
 ---
 

docs/left_scheduler_work.md

@@ -1,7 +1,7 @@
 # Scheduler 剩余工作清单（left_scheduler_work）
 
 - 状态：持续更新
-- 日期：2026-08-07
+- 日期：2026-08-13
 - 来源：项目自带 MD 设计文档分析（.kiro/steering、.kiro/specs、docs/、CLAUDE.md）
 
 本文汇总 scheduler 相关的一切剩余/未完成/已知损坏的工作，按优先级排序。每项标注来源文档与状态。
@@ -155,10 +155,18 @@ spec 声称 36/36 任务完成，但 re-audit（research.md）发现缺陷。来
 |-------|------|------|
 | Phase 0 | 只读 MCP Server：6 读工具、schedulerclient、`mcp_readonly` 角色、API Key 认证 | ✅ 2026-07-26 |
 | Phase 1 | 4 写工具（submit/cancel/hold/release）、`general_serial` 模板、`mcp_operator`、`dry_run` 审查 | ✅ 2026-08-01（代码）；**2026-08-12 完整集群 E2E 验证通过**：submit→运行→DONE、cancel(kill)→EXIT、读接口全通。注意：job control 动作是 kill/stop/resume/checkpoint（schedulerclient 已正确映射） |
+| Phase 2 | SSE 事件流（`/xce/v1/events` + EventBroker）+ `WatchJobs()` + 4 高级工具 | ✅ **2026-08-12 实施并集群验证**（见下）|
+
+**Phase 2 落地明细（2026-08-12）**：
+- **apiserver**：`/xce/v1/events` SSE endpoint + `backend.Broker`（EventBroker，DefaultHandler 发布 job.created/updated/deleted/finished）；RBAC `event:view`（mcp_readonly/mcp_operator/cluster_admin）；whitelist 加 `/xce/v1/events`。
+- **schedulerclient**：`WatchJobs(ctx)` 订阅 SSE 流；**顺带修复** `ListJobs` 的 filter 参数——apiserver 用 `filter=queue_name:eq:...`（field:op:value），原 `queue_name`/`submit_user` 参数被 400 拒绝（影响带过滤的列表 + estimate_wait_time）。
+- **mcp-server**：14 工具（10 原有 + `scheduler_wait_job`/`scheduler_analyze_failure`/`scheduler_suggest_resources`/`scheduler_estimate_wait_time`）。
+- **人类可读 jobid（2026-08-13）**：apiserver 响应加 `job_id_human`/`human_id`（`cluster-apiserver-时间戳-序号`，如 `1-0-1786585485-0`），复用 `backend.FormatJobID`——MCP 原来只返回裸 64 位数字（`73930964699447296`），xce_test/xcecli 只认连字符格式，导致跨工具无法引用。架构原则：**apiserver 负责数据正确性，mcp-server 只读取透传**。
+- **Claude Code 集成**：`.mcp.json` 注册 `xce-scheduler` server，key 用 `${XCE_API_KEY}` 环境变量（不硬编码提交）；apiserver 重启后 key 失效需重新生成。
 
 **注意**：apiserver 在 controller 重启后**不会自动重新 join**（只重连 TCP，DataStore 保持陈旧为空）——controller 重启后需手动重启 apiserver 才会重新收到作业推送。这与 scheduler/worker 的重连缺口同类，属 component-failover-adoption 待办。
 
-代码：`src/frontend/mcp-server/`（10 工具，~1300 行）、`src/frontend/schedulerclient/`（45 测试）、apiserver API Key + RBAC 支持。
+代码：`src/frontend/mcp-server/`（14 工具）、`src/frontend/schedulerclient/`（~50 测试）、apiserver API Key + RBAC + SSE 支持。
 
 ### 1. 阻塞性 bug（影响 chatbot E2E 验证）
 | Bug | 影响 |
@@ -167,9 +175,10 @@ spec 声称 36/36 任务完成，但 re-audit（research.md）发现缺陷。来
 | **bug-1494** | 作业全 PEND 不调度，Agent 提交流程无法完成。**已修复（2026-08-11）**：子问题一 dispatch-fail 重试（bug-1510）+ 子问题二 worker 重新 join 后 slots 不更新（见第一节） |
 | **chatbot 过度建议** | ✅ 已缓解（3db216c）：mcp-server 加 server instructions + 状态工具 report-only 描述，防止 LLM 把非 OK 状态当事故、主动给一堆排查建议 |
 
-### 2. Phase 2：监控 + 纠错闭环（未开始，估 2-3 天）
-- **SSE 事件流**（ADR-004 设计已完、未实现）：apiserver SSE endpoint + EventBroker，schedulerclient `WatchJobs()`，mcp-server 事件接入。Agent 工作流：submit → watch → analyze → auto-handle
-- 3 个高级工具：`scheduler_analyze_failure`、`scheduler_suggest_resources`、`scheduler_estimate_wait_time`
+### 2. Phase 2：监控 + 纠错闭环（✅ 已实施 2026-08-12）
+- **SSE 事件流**（ADR-004 已实施）：apiserver `/xce/v1/events` + EventBroker、schedulerclient `WatchJobs()`、mcp-server 4 高级工具。t1 集群验证：SSE 流推送 PEND→RUN→DONE、WatchJobs 捕获完整生命周期、4 工具全部 live 工作。
+- 高级工具：`scheduler_wait_job`、`scheduler_analyze_failure`、`scheduler_suggest_resources`、`scheduler_estimate_wait_time`（14 工具）
+- Agent 工作流已打通：submit → wait_job → 取日志 → analyze（详见设计文档 §5.3）
 
 ### 3. Phase 3 / Phase 4：后端适配器（未开始）
 - Phase 3：**LSF adapter**（CLI 解析优先，Python Sidecar 后续，估 +1 月）
@@ -186,11 +195,12 @@ spec 声称 36/36 任务完成，但 re-audit（research.md）发现缺陷。来
 
 ---
 
-## 建议优先级
+## 建议优先级（2026-08-13 更新）
 
-1. **bug-1490**（apiserver control HTTP 500）——阻塞 chatbot 写工具 E2E，小而明确，能立即修。
-2. **bug-1494**（作业 PEND）——根因已定位（worker+scheduler 不自愈）。G-8 死代码（scheduler 作业更新路径损坏）很可能是元凶。
-3. **msg-gap G-8/G-3/G-5**——小而明确，能立即修。
-4. **component-failover-adoption**（scheduler 重连）——与 bug-1494 的 join 链路疑点直接相关。
-5. scheduler-agent Phase 2（SSE 事件流）——依赖 bug-1490/bug-1494 先修好。
-6. scheduler-framework 重构——大规划，建议等前 5 项稳定后再动。
+1. ~~**bug-1490**~~ ✅ 已修复（c90e63e），写工具 E2E 通过。
+2. ~~**bug-1494**~~ ✅ 已修复（bug-1510 + bug-1511，2026-08-11），作业全 PEND 解决。
+3. ~~**scheduler-agent Phase 2（SSE 事件流）**~~ ✅ 已实施并验证（2026-08-12）。
+4. **msg-gap G-8/G-3/G-5**——scheduler-controller 消息缺陷，小而明确，能立即修。
+5. **component-failover-adoption**（scheduler/worker/apiserver 重启后自动重连）——仍是最重要基础设施缺口；apiserver 的 rejoin 缺口已实测确认。
+6. **scheduler-framework 重构**——大规划，建议等前几项稳定后再动。
+7. **scheduler-agent 下一步**：Phase 3 LSF adapter（估 +1 月），或先跑通 agent 工作流真实试用 + 补齐 3 个未决决策（LLM 选型/多集群 URM/air-gapped）。