lei.chen 4e894d2184 merge: sync Rongsong's scheduler-framework into chatbot

Full merge of origin/scheduler-framework (scheduler-core restructure,
failover-adoption, WAL_JOB_DISPATCH, apiserver backend health/heartbeat,
build-time versioning, bug-1673/1676/1678/1679/1680).

Conflict resolution:
- scheduler: adopt Rongsong's scheduler-core architecture (driver loop,
  queue project/picker fields, resource bug-1569 fix)
- worker/controller_connector.c: keep Rongsong's reconnect/rejoin
  structure, PORT our bug-1509 resend_job_status onto the rejoin success
  path (his reconnect didn't re-report job state)
- apiserver backend: take Rongsong's health/heartbeat (supersedes our
  e505066 circuit breaker)
- group.c: keep our bug-1513 comment
- misc: take theirs for additive fixes (go_build versioning,
  test_group.c, persist.c RUSAGE_UPDATE, FilterRows bug-1680)

NOTE: our scheduler fixes (bug-1494 family dispatch-fail requeue,
PEND-reason classification) are NOT in Rongsong's branch — needs
verification/porting onto scheduler-core (separate follow-up).

Co-Authored-By: Claude <noreply@anthropic.com>

2026-08-25 12:33:16 +00:00

7 KiB

Raw Blame History

OpenWolf

@.wolf/OPENWOLF.md

This project uses OpenWolf for context management. Read and follow .wolf/OPENWOLF.md every session. Check .wolf/cerebrum.md before generating code. Check .wolf/anatomy.md before reading files.

XCE Cluster Management System

Project Overview

XCE is an HPC cluster management system — Go API/CLI tier fronting C daemons (controller, worker, scheduler). Full context in .kiro/steering/product.md.

Engineering Guide

For details on any topic below, read the corresponding steering file in .kiro/steering/. CLAUDE.md provides pointers, not duplicates.

Topic	Steering File
Project purpose, HA phases, design decisions	`product.md`
Technology stack, build systems, key decisions	`tech.md`
Directory layout, naming conventions, test org	`structure.md`
Build commands, Nix flake, deploy, AppImage	`build-deployment.md`
REST handler pattern, daemon opcodes, Cap'n Proto	`api-standards.md`
HA protocol, async I/O, Raft election, heartbeat	`ha-async-io.md`
TDD rules, C + Go test patterns, coverage targets	`testing.md`
Error handling: Go ReturnError, C errnox, zap logging	`error-handling.md`
RBAC, JWT, auth backends, input validation, secrets	`security.md`

Key Rules (from steering — summarized for quick reference)

TDD mandatory: red → green → refactor; test in testsuite/ before C changes, _test.go before Go changes
Async I/O: delaymgr is trigger-only and must never block (no connect(), no file I/O) — it drives heartbeats/election timeouts, so blocking there causes spurious failover. Long/unbounded blocking goes on the slow lane; bounded blocking in vtask handlers is permitted only under an admission reservation (COMMIT_WAIT_RESERVED). xce_request/xce_reply are always non-blocking. See ha-async-io.md
Connections are long-lived: closing signals RDHUP failover. Inbound connections are vote-only; outbound carry all state push
Middleware chain (non-negotiable): WhitelistURL → WhitelistIP → RateLimit → JWT → CORS → RBAC
Error pattern: ReturnError(code, msg, nil); return — early return on every error
Build: /xce-build [component] — debug + ASAN + tests. See build-deployment.md for full commands

Key Files

src/frontend/Makefile — primary build/test orchestration
src/frontend/apiserver/cmd/main.go — server startup, init sequence
src/frontend/apiserver/router/routers.go — all routes and middleware chain
src/frontend/apiserver/controllers/rbac_check.go — RBAC loading and enforcement
src/frontend/apiserver/config/rbac.yaml — role/permission definitions
src/frontend/apiserver/config/auth.yaml — auth provider selection

Subdirectory Guidelines

src/frontend/ — See src/frontend/CLAUDE.md
src/controller/ — See src/controller/CLAUDE.md
src/worker/ — See src/worker/CLAUDE.md
src/scheduler/ — See src/scheduler/CLAUDE.md
src/lib/ — See src/lib/CLAUDE.md
src/frontend/schedulerclient/, src/frontend/mcp-server/ — Phase 0-1 已完成（Phase 0: 2026-07-26，Phase 1: 2026-08-01）。schedulerclient 是独立 Go library，提供读 + 写 Client（6 只读 + 4 写：SubmitJob/CancelJob/HoldJob/ReleaseJob，resty HTTP 实现，33+12 单测）。mcp-server 是 MCP stdio 二进制（github.com/mark3labs/mcp-go@v0.57.0），通过 schedulerclient 调用 apiserver REST API，暴露 10 个 MCP Tool（6 只读 + 4 写，写工具含 dry_run review 模式 + general_serial 作业模板 registry）。Pure Go, CGO_ENABLED=0。Phase 1 需 mcp_operator RBAC 角色（config/rbac.yaml）。总体架构与分阶段计划见 docs/Scheduler-Agent-设计文档.md。集群验证（2026-08-01/02）：t1 单节点 controller 集群上，读操作（含 OrderMap 解析修复 bug-1491）+ submit 已通过 mcp_operator API key 验证；作业全部 PEND 不调度（bug-1494，疑 scheduler→controller join 链路，已补 resources.yaml 但未解决）；hold/release/cancel 因 apiserver control 链路预存 bug（HTTP 500）待修（bug-1490）。

Agentic SDLC and Spec-Driven Development

Kiro-style Spec-Driven Development on an agentic SDLC

Project Context

Paths

Steering: .kiro/steering/
Specs: .kiro/specs/

Steering vs Specification

Steering (.kiro/steering/) - Guide AI with project-wide rules and context Specs (.kiro/specs/) - Formalize development process for individual features

Active Specifications

Check .kiro/specs/ for active specifications
Use /kiro-spec-status [feature-name] to check progress

Development Guidelines

Think in English, generate responses in English. All Markdown content written to project files (e.g., requirements.md, design.md, tasks.md, research.md, validation reports) MUST be written in the target language configured for this specification (see spec.json.language).

Minimal Workflow

Phase 0 (optional): /kiro-steering, /kiro-steering-custom
Phase 1 (Specification):
- /kiro-spec-init "description" — initialize spec scaffold
- /kiro-spec-quick "description" — fast single-pass spec (requirements + design + tasks, no approval gates)
- /kiro-spec-batch — generate specs for all features in a roadmap in parallel
- /kiro-spec-requirements {feature}
- /kiro-validate-gap {feature} (optional: for existing codebase)
- /kiro-spec-design {feature} [-y]
- /kiro-validate-design {feature} (optional: design review)
- /kiro-spec-tasks {feature} [-y]
Phase 2 (Implementation): /kiro-impl {feature} [tasks]
- /kiro-validate-impl {feature} (optional: after implementation)
- /kiro-verify-completion {feature} — validate all acceptance criteria are met
Debugging: /kiro-debug {feature} — systematic debugging with spec context
Code review: /kiro-review — review implementation against spec requirements
Progress check: /kiro-spec-status {feature} (use anytime)

Development Rules

3-phase approval workflow: Requirements → Design → Tasks → Implementation
Human review required each phase; use -y only for intentional fast-track
Use /kiro-spec-quick for small, well-understood features; full workflow for anything touching HA/persistence/RBAC
Keep steering current and verify alignment with /kiro-spec-status
Follow the user's instructions precisely, and within that scope act autonomously: gather the necessary context and complete the requested work end-to-end in this run, asking questions only when essential information is missing or the instructions are critically ambiguous.

Steering Configuration

Load entire .kiro/steering/ as project memory — all .md files are authoritative
Core: product.md, tech.md, structure.md
Custom: testing.md, error-handling.md, security.md, ha-async-io.md, api-standards.md, build-deployment.md, config-design.md, persistence.md, roadmap.md
Add new custom files via /kiro-steering-custom; sync existing via /kiro-steering

7 KiB Raw Blame History Unescape Escape