myxce/CLAUDE.md

OpenWolf

@.wolf/OPENWOLF.md

This project uses OpenWolf for context management. Read and follow .wolf/OPENWOLF.md every session. Check .wolf/cerebrum.md before generating code. Check .wolf/anatomy.md before reading files.

XCE Cluster Management System

Overview

XCE is an HPC cluster management system providing job scheduling, queue management, and cluster resource control. The Go-based API/CLI tier fronts C system daemons (controller, worker, scheduler).

Tech Stack

• Go 1.23.4: REST API server (src/frontend/apiserver), CLI tool (src/frontend/xcecli)
• C (gnu17): Core daemons — controller, worker, scheduler, shared lib
• Build: Meson + Nix flakes for C components; make for Go components
• Go key deps: Gin v1.10.0, JWT v4, Zap logging, Viper config, Cobra CLI

Development Commands

# Build with /xce-build skill (debug + ASAN, clean build, runs tests)
/xce-build                  # Build all components + run all tests
/xce-build lib              # Build libxce + run lib tests
/xce-build controller       # Build lib + controller + run controller tests
/xce-build scheduler        # Build lib + scheduler + run scheduler tests
/xce-build worker           # Build lib + worker (no unit tests registered)
/xce-build apiserver        # Build lib + apiserver + xcecli + run Go unit tests
/xce-build xcecli           # Build xcecli only + run xcecli Go tests

Test-Driven Development

Follow TDD strictly throughout the full development process:

1. Red — Write a failing test that specifies the desired behaviour before writing any implementation code.
2. Green — Write the minimal implementation to make the test pass. Do not over-engineer.
3. Refactor — Clean up the implementation and tests while keeping all tests green.

Rules

• Never implement a feature without a corresponding test written first.
• Tests must be runnable without a live server or external services (use table-driven tests, stubs, and t.TempDir() for fixtures).
• A change is not complete until make test (and make test-integration where applicable) passes.
• For C components, add a test case in the relevant testsuite/ before modifying the implementation.
• Tests live next to the code they cover (_test.go for Go, testsuite/ for C).
• Keep test scope narrow — unit-test business logic; use integration tests for HTTP/API surface.

Architecture

Request → Gin Router → Middleware (CORS/RateLimit/IPWhitelist/JWT) → RBAC Check → Controller → Services → In-Memory Store

• 3 independent Go modules: apiserver, xcecli, bcctool (no go.work)
• Stateful API server: in-memory stores seeded at startup via store.LoadBuiltinData()
• Pluggable auth: configfile / LDAP / OAuth2 / PAM (set in config/auth.yaml)
• RBAC: domain-based role-action-domain model loaded from config/rbac.yaml

Connection Model and Async I/O

Long-Lived Persistent Connections (CRITICAL)

All XCE daemon connections are long-lived — they are established once and reused for the lifetime of the role relationship:

Connection	Direction	Owner	Purpose
Controller primary → standby	Primary dials out during election	Primary (ha->standby_endpoints)	Config push, WAL replication, heartbeat
Controller standby ← primary	Inbound on standby listener	Standby (primary_ctrl_fd)	Receives ctrlhello, WAL, config, heartbeat
Primary → worker	Primary dials out	Primary (c->workers[i].ep)	Job dispatch, heartbeat
Scheduler → primary	Scheduler dials out	Scheduler	Schedule requests
API server → primary	API server dials out	API server backend	Job submit, cluster queries

Connections are never closed and re-opened between RPCs. Closing a connection signals a failure event (RDHUP) which triggers failover or reconnect logic.

Async Non-Blocking I/O Model (CRITICAL)

All C daemon I/O runs through iomgr (epoll-based) + vtaskmgr (thread pool):

• iomgr — single-threaded epoll loop; dispatches inbound packets as vtasks; manages endpoint lifetime
• vtaskmgr — thread-pool executing vtask handlers; handlers must return quickly (no blocking I/O)
• delaymgr — timer thread for periodic/delayed tasks; the only place blocking operations (connect, file I/O) are permitted
• xce_request(ep, &req) — enqueues a packet to the endpoint's send buffer; always non-blocking
• xce_reply(ep, host, &reply) — enqueues a reply; always non-blocking

Never block in a vtask handler. Blocking calls (connect, read, usleep) belong in delaymgr tasks or dedicated threads.

Primary Heartbeat to Standbys

The primary sends OPCode_nop to each standby every heartbeat_interval_ms via a self-perpetuating delaymgr one-shot task (ha_start_primary_heartbeat). The task re-arms itself only while role is PRIMARY or PRIMARY_COMMITTING — on demotion the loop stops automatically (no cancel API needed). This keeps the standby-side watchdog (ha_arm_heartbeat_timeout_vtask) from timing out during idle periods.

Key Files

• src/frontend/Makefile — primary build/test orchestration
• src/frontend/apiserver/cmd/main.go — server startup, init sequence
• src/frontend/apiserver/router/routers.go — all routes and middleware chain
• src/frontend/apiserver/controllers/rbac_check.go — RBAC loading and enforcement
• src/frontend/apiserver/config/rbac.yaml — role/permission definitions
• src/frontend/apiserver/config/auth.yaml — auth provider selection

Subdirectory Guidelines

• src/frontend/ — See src/frontend/CLAUDE.md
• src/controller/ — See src/controller/CLAUDE.md
• src/worker/ — See src/worker/CLAUDE.md
• src/scheduler/ — See src/scheduler/CLAUDE.md
• src/lib/ — See src/lib/CLAUDE.md

Agentic SDLC and Spec-Driven Development

Kiro-style Spec-Driven Development on an agentic SDLC

Project Context

Paths

• Steering: .kiro/steering/
• Specs: .kiro/specs/

Steering vs Specification

Steering (.kiro/steering/) - Guide AI with project-wide rules and context Specs (.kiro/specs/) - Formalize development process for individual features

Active Specifications

• Check .kiro/specs/ for active specifications
• Use /kiro-spec-status [feature-name] to check progress

Development Guidelines

• Think in English, generate responses in English. All Markdown content written to project files (e.g., requirements.md, design.md, tasks.md, research.md, validation reports) MUST be written in the target language configured for this specification (see spec.json.language).

Minimal Workflow

• Phase 0 (optional): /kiro-steering, /kiro-steering-custom
• Phase 1 (Specification):
  • /kiro-spec-init "description" — initialize spec scaffold
  • /kiro-spec-quick "description" — fast single-pass spec (requirements + design + tasks, no approval gates)
  • /kiro-spec-batch — generate specs for all features in a roadmap in parallel
  • /kiro-spec-requirements {feature}
  • /kiro-validate-gap {feature} (optional: for existing codebase)
  • /kiro-spec-design {feature} [-y]
  • /kiro-validate-design {feature} (optional: design review)
  • /kiro-spec-tasks {feature} [-y]
• Phase 2 (Implementation): /kiro-impl {feature} [tasks]
  • /kiro-validate-impl {feature} (optional: after implementation)
  • /kiro-verify-completion {feature} — validate all acceptance criteria are met
• Debugging: /kiro-debug {feature} — systematic debugging with spec context
• Code review: /kiro-review — review implementation against spec requirements
• Progress check: /kiro-spec-status {feature} (use anytime)

Development Rules

• 3-phase approval workflow: Requirements → Design → Tasks → Implementation
• Human review required each phase; use -y only for intentional fast-track
• Use /kiro-spec-quick for small, well-understood features; full workflow for anything touching HA/persistence/RBAC
• Keep steering current and verify alignment with /kiro-spec-status
• Follow the user's instructions precisely, and within that scope act autonomously: gather the necessary context and complete the requested work end-to-end in this run, asking questions only when essential information is missing or the instructions are critically ambiguous.

Steering Configuration

• Load entire .kiro/steering/ as project memory
• Core files: product.md, tech.md, structure.md
• XCE custom files: testing.md, error-handling.md, security.md
• Add new custom files via /kiro-steering-custom