myxce/CLAUDE.md

OpenWolf

@.wolf/OPENWOLF.md

This project uses OpenWolf for context management. Read and follow .wolf/OPENWOLF.md every session. Check .wolf/cerebrum.md before generating code. Check .wolf/anatomy.md before reading files.

XCE Cluster Management System

Overview

XCE is an HPC cluster management system providing job scheduling, queue management, and cluster resource control. The Go-based API/CLI tier fronts C system daemons (controller, worker, scheduler).

Tech Stack

• Go 1.23.4: REST API server (src/frontend/apiserver), CLI tool (src/frontend/xcecli)
• C (gnu17): Core daemons — controller, worker, scheduler, shared lib
• Build: Meson + Nix flakes for C components; make for Go components
• Go key deps: Gin v1.10.0, JWT v4, Zap logging, Viper config, Cobra CLI

Development Commands

# Build with /xce-build skill (debug + ASAN, clean build, runs tests)
/xce-build                  # Build all components + run all tests
/xce-build lib              # Build libxce + run lib tests
/xce-build controller       # Build lib + controller + run controller tests
/xce-build scheduler        # Build lib + scheduler + run scheduler tests
/xce-build worker           # Build lib + worker (no unit tests registered)
/xce-build apiserver        # Build lib + apiserver + xcecli + run Go unit tests
/xce-build xcecli           # Build xcecli only + run xcecli Go tests

# From src/frontend/ (manual Go dev loop)
make build-server       # Build apiserver binary → /tmp/apiserver
make build-cli          # Build xcecli binary → /tmp/xce_static (CGO_ENABLED=0)
make test               # Unit tests (no live server required)
make test-integration   # Regression suite (requires XCE_BIN and SRV_URL env vars)
make test-all           # Unit + integration tests

# Nix package builds (from src/)
nix build --option sandbox false .#xcecore       # C daemons + frontend/apiserver
nix build --option sandbox false .#xcecore_debug # Same with debug/ASAN
nix build --option sandbox false .#xcecli        # CLI tool only

Test-Driven Development

Follow TDD strictly throughout the full development process:

1. Red — Write a failing test that specifies the desired behaviour before writing any implementation code.
2. Green — Write the minimal implementation to make the test pass. Do not over-engineer.
3. Refactor — Clean up the implementation and tests while keeping all tests green.

Rules

• Never implement a feature without a corresponding test written first.
• Tests must be runnable without a live server or external services (use table-driven tests, stubs, and t.TempDir() for fixtures).
• A change is not complete until make test (and make test-integration where applicable) passes.
• For C components, add a test case in the relevant testsuite/ before modifying the implementation.
• Tests live next to the code they cover (_test.go for Go, testsuite/ for C).
• Keep test scope narrow — unit-test business logic; use integration tests for HTTP/API surface.

Architecture

Request → Gin Router → Middleware (CORS/RateLimit/IPWhitelist/JWT) → RBAC Check → Controller → Services → In-Memory Store

• 3 independent Go modules: apiserver, xcecli, bcctool (no go.work)
• Stateful API server: in-memory stores seeded at startup via store.LoadBuiltinData()
• Pluggable auth: configfile / LDAP / OAuth2 / PAM (set in config/auth.yaml)
• RBAC: domain-based role-action-domain model loaded from config/rbac.yaml

Connection Model and Async I/O

Long-Lived Persistent Connections (CRITICAL)

All XCE daemon connections are long-lived — they are established once and reused for the lifetime of the role relationship:

Connection	Direction	Owner	Purpose
Controller primary → standby	Primary dials out during election	Primary (ha->standby_endpoints)	Config push, WAL replication, heartbeat
Controller standby ← primary	Inbound on standby listener	Standby (primary_ctrl_fd)	Receives ctrlhello, WAL, config, heartbeat
Primary → worker	Primary dials out	Primary (c->workers[i].ep)	Job dispatch, heartbeat
Scheduler → primary	Scheduler dials out	Scheduler	Schedule requests
API server → primary	API server dials out	API server backend	Job submit, cluster queries

Connections are never closed and re-opened between RPCs. Closing a connection signals a failure event (RDHUP) which triggers failover or reconnect logic.

Async Non-Blocking I/O Model (CRITICAL)

All C daemon I/O runs through iomgr (epoll-based) + vtaskmgr (thread pool):

• iomgr — single-threaded epoll loop; dispatches inbound packets as vtasks; manages endpoint lifetime
• vtaskmgr — thread-pool executing vtask handlers; handlers must return quickly (no blocking I/O)
• delaymgr — timer thread for periodic/delayed tasks; the only place blocking operations (connect, file I/O) are permitted
• xce_request(ep, &req) — enqueues a packet to the endpoint's send buffer; always non-blocking
• xce_reply(ep, host, &reply) — enqueues a reply; always non-blocking

Never block in a vtask handler. Blocking calls (connect, read, usleep) belong in delaymgr tasks or dedicated threads.

Primary Heartbeat to Standbys

The primary sends OPCode_nop to each standby every heartbeat_interval_ms via a self-perpetuating delaymgr one-shot task (ha_start_primary_heartbeat). The task re-arms itself only while role is PRIMARY or PRIMARY_COMMITTING — on demotion the loop stops automatically (no cancel API needed). This keeps the standby-side watchdog (ha_arm_heartbeat_timeout_vtask) from timing out during idle periods.

Key Files

• src/frontend/Makefile — primary build/test orchestration
• src/frontend/apiserver/cmd/main.go — server startup, init sequence
• src/frontend/apiserver/router/routers.go — all routes and middleware chain
• src/frontend/apiserver/controllers/rbac_check.go — RBAC loading and enforcement
• src/frontend/apiserver/config/rbac.yaml — role/permission definitions
• src/frontend/apiserver/config/auth.yaml — auth provider selection

Subdirectory Guidelines

• src/frontend/ — See src/frontend/CLAUDE.md
• src/controller/ — See src/controller/CLAUDE.md
• src/worker/ — See src/worker/CLAUDE.md
• src/scheduler/ — See src/scheduler/CLAUDE.md
• src/lib/ — See src/lib/CLAUDE.md

MCP Tools: code-review-graph

IMPORTANT: This project has a knowledge graph. ALWAYS use the code-review-graph MCP tools BEFORE using Grep/Glob/Read to explore the codebase. The graph is faster, cheaper (fewer tokens), and gives you structural context (callers, dependents, test coverage) that file scanning cannot.

When to use graph tools FIRST

• Exploring code: semantic_search_nodes or query_graph instead of Grep
• Understanding impact: get_impact_radius instead of manually tracing imports
• Code review: detect_changes + get_review_context instead of reading entire files
• Finding relationships: query_graph with callers_of/callees_of/imports_of/tests_for
• Architecture questions: get_architecture_overview + list_communities

Fall back to Grep/Glob/Read only when the graph doesn't cover what you need.

Key Tools

Tool	Use when
detect_changes	Reviewing code changes — gives risk-scored analysis
get_review_context	Need source snippets for review — token-efficient
get_impact_radius	Understanding blast radius of a change
get_affected_flows	Finding which execution paths are impacted
query_graph	Tracing callers, callees, imports, tests, dependencies
semantic_search_nodes	Finding functions/classes by name or keyword
get_architecture_overview	Understanding high-level codebase structure
refactor_tool	Planning renames, finding dead code

Workflow

1. The graph auto-updates on file changes (via hooks).
2. Use detect_changes for code review.
3. Use get_affected_flows to understand impact.
4. Use query_graph pattern="tests_for" to check coverage.

graphify

This project has a graphify knowledge graph at graphify-out/.

Rules:

• Before answering architecture or codebase questions, read graphify-out/GRAPH_REPORT.md for god nodes and community structure
• If graphify-out/wiki/index.md exists, navigate it instead of reading raw files
• For cross-module "how does X relate to Y" questions, prefer graphify query "<question>", graphify path "<A>" "<B>", or graphify explain "<concept>" over grep — these traverse the graph's EXTRACTED + INFERRED edges instead of scanning files
• After modifying code files in this session, run graphify update . to keep the graph current (AST-only, no API cost)

Agentic SDLC and Spec-Driven Development

Kiro-style Spec-Driven Development on an agentic SDLC

Project Context

Paths

• Steering: .kiro/steering/
• Specs: .kiro/specs/

Steering vs Specification

Steering (.kiro/steering/) - Guide AI with project-wide rules and context Specs (.kiro/specs/) - Formalize development process for individual features

Active Specifications

• Check .kiro/specs/ for active specifications
• Use /kiro-spec-status [feature-name] to check progress

Development Guidelines

• Think in English, generate responses in English. All Markdown content written to project files (e.g., requirements.md, design.md, tasks.md, research.md, validation reports) MUST be written in the target language configured for this specification (see spec.json.language).

Minimal Workflow

• Phase 0 (optional): /kiro-steering, /kiro-steering-custom
• Phase 1 (Specification):
  • /kiro-spec-init "description" — initialize spec scaffold
  • /kiro-spec-quick "description" — fast single-pass spec (requirements + design + tasks, no approval gates)
  • /kiro-spec-batch — generate specs for all features in a roadmap in parallel
  • /kiro-spec-requirements {feature}
  • /kiro-validate-gap {feature} (optional: for existing codebase)
  • /kiro-spec-design {feature} [-y]
  • /kiro-validate-design {feature} (optional: design review)
  • /kiro-spec-tasks {feature} [-y]
• Phase 2 (Implementation): /kiro-impl {feature} [tasks]
  • /kiro-validate-impl {feature} (optional: after implementation)
  • /kiro-verify-completion {feature} — validate all acceptance criteria are met
• Debugging: /kiro-debug {feature} — systematic debugging with spec context
• Code review: /kiro-review — review implementation against spec requirements
• Progress check: /kiro-spec-status {feature} (use anytime)

Development Rules

• 3-phase approval workflow: Requirements → Design → Tasks → Implementation
• Human review required each phase; use -y only for intentional fast-track
• Use /kiro-spec-quick for small, well-understood features; full workflow for anything touching HA/persistence/RBAC
• Keep steering current and verify alignment with /kiro-spec-status
• Follow the user's instructions precisely, and within that scope act autonomously: gather the necessary context and complete the requested work end-to-end in this run, asking questions only when essential information is missing or the instructions are critically ambiguous.

Steering Configuration

• Load entire .kiro/steering/ as project memory
• Core files: product.md, tech.md, structure.md
• XCE custom files: testing.md, error-handling.md, security.md
• Add new custom files via /kiro-steering-custom