claude skills

.kiro/steering/error-handling.md

@@ -0,0 +1,92 @@
+# Error Handling Standards
+
+## Philosophy
+
+Fail fast with a consistent, machine-readable response. Handle errors at the controller boundary; let business logic return typed errors. Never silently swallow errors.
+
+## Go API Response Pattern
+
+All HTTP responses go through two helpers in `controllers/response.go`:
+
+```go
+// Success (always HTTP 200)
+ReturnSuccess(ctx, common.SUCCESS_CODE, common.SUCCESS_MSG, data)
+
+// Error (HTTP status == code parameter)
+ReturnError(ctx, common.ERROR_AUTH_CODE, common.ERROR_ACCESS_FORBIDDEN_MSG, nil)
+ctx.Abort()   // always Abort() after ReturnError in middleware
+return        // always return immediately after ReturnError in handlers
+```
+
+**The early-return pattern is non-negotiable.** Every error path ends with `ReturnError(...); return`.
+
+## HTTP Status Code Mapping
+
+| Code | Constant | When to use |
+|------|----------|-------------|
+| 200 | `SUCCESS_CODE` | Successful operation |
+| 201 | `CREATED` | Resource created |
+| 400 | `INVALID_PARAMS_CODE` | Malformed or invalid input |
+| 401 | `ERROR_AUTH_CHECK_TOKEN_CODE` | Missing or invalid JWT |
+| 403 | `ERROR_AUTH_CODE` | Valid JWT but insufficient permission |
+| 404 | `ERROR_NOT_FOUND_CODE` | Resource not found |
+| 429 | `ERROR_TOO_MANY_REQUESTS_CODE` | Rate limit exceeded |
+| 500 | `ERROR_CODE` | Unexpected server error |
+
+All code constants are defined in `common/common.go`. Add new codes there — do not inline numeric literals in handlers.
+
+## Structured Logging with Zap
+
+Log at the call site before returning an error. Use typed field helpers — never `fmt.Sprintf` in zap calls.
+
+```go
+// Preferred pattern
+logger.Error("rate limit exceeded",
+    zap.String("path", path),
+    zap.String("client_ip", ctx.ClientIP()),
+    zap.Int("errorcode", code),
+)
+
+// For error values
+logger.Error("validate token error", zap.Error(err))
+
+// For complex types
+logger.Debug("permissions resolved", zap.Any("permsWithType", permsWithType))
+```
+
+**Log levels**:
+- `Error` — operation failed, client or server fault
+- `Warn` — recoverable edge case (e.g., retried operation succeeded)
+- `Info` — key lifecycle events (server start, config load)
+- `Debug` — diagnostic detail, disabled in production
+
+**Never log**: JWT tokens, passwords, secret keys, or full request bodies containing sensitive data.
+
+## C Error Propagation
+
+Return negative errno-style codes for C daemon errors. Use the `errnox.h` definitions from `src/include/xce/`. Check and propagate immediately — do not continue on error:
+
+```c
+int rc = xce_send_request(&req);
+if (rc < 0) {
+    log_error("send_request failed: %d", rc);
+    return rc;
+}
+```
+
+## Input Validation Pattern
+
+Validate at the controller boundary using a `preCheck{Resource}Request()` helper before touching services:
+
+```go
+func (c JobController) SubmitJob(ctx *gin.Context) {
+    req, err := preCheckSubmitJobRequest(ctx)
+    if err != nil {
+        ReturnError(ctx, common.INVALID_PARAMS_CODE, err.Error(), nil)
+        return
+    }
+    // ... service call
+}
+```
+
+Keep `preCheck*` functions in the same file as the handler. They bind, validate, and return the typed request struct or an error.

.kiro/steering/product.md

@@ -0,0 +1,43 @@
+# Product Overview
+
+XCE is an HPC (High-Performance Computing) cluster management system providing job scheduling, queue management, and cluster resource control. It targets organizations running batch workloads on on-premises or air-gapped compute clusters who need deterministic scheduling, multi-tenant access control, and zero external database dependencies.
+
+## Core Capabilities
+
+1. **Job lifecycle management** — submit, start, stop, restart, kill, and monitor jobs including job arrays. Resource usage data is collected and exposed via API.
+2. **Queue and scheduling** — plugin-based scheduling engine for job-to-node placement with queue control (enable/disable) and expression-based resource filtering.
+3. **Cluster and host management** — node status monitoring (online/offline/unavailable), cluster topology tracking, and host control operations.
+4. **Multi-tenant RBAC + pluggable auth** — domain-based `action:scope` permission model (e.g., `job:view:self`). Auth backends: configfile, LDAP, OAuth2, PAM. Built-in IP whitelist and rate limiting.
+5. **Controller persistence and HA** — WAL + checkpoint-based single-controller persistence with LSF-style priority election for standby failover and live replication over TCP (Cap'n Proto serialization).
+
+## Target Use Cases
+
+- **Local HPC batch computing** — long-running scientific/engineering workloads requiring deterministic scheduling, queue priority, and resource isolation.
+- **Air-gapped / secure environments** — standalone deployments with file-based auth + SSL, no external cloud dependencies.
+- **Multi-team shared clusters** — multiple teams sharing a cluster through RBAC with JWT token authentication.
+- **No-shared-storage HPC** — controller uses local filesystem (ext4/xfs) for storage; job files transferred via API upload/download rather than network shares.
+
+## Value Proposition
+
+XCE provides a **fully integrated single-vendor stack** (no database setup, unified build system) with an open, buildable codebase (Nix flakes, AppImage packaging). Compared to Slurm/PBS, it eliminates the external database requirement. Compared to commercial LSF, it is open source and reproducibly buildable.
+
+**Key architectural trade-off**: single-writer model throughout — no multi-master consensus. HA failover uses LSF-style priority election (simple, predictable) rather than Raft (10× complexity). All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
+
+## Active Development Context
+
+The project is in a **persistence and HA transformation** (branch `persistent_ha`). Phases:
+
+| Phase | Goal | Status |
+|-------|------|--------|
+| P0 | Restart survival (WAL + checkpoint) | Implementing |
+| P1 | Standby election and takeover | Designed |
+| P2 | Live replication primary → standby | Designed |
+| P2.5 | Worker/scheduler auto-reconnect on failover | Designed |
+| P3–P4 | Hardening, metrics, docs | Planned |
+
+Key design decisions already locked:
+- No `fork()` for snapshots (mutex inheritance deadlock in multithreaded controller)
+- No Raft (single writer makes consensus overhead unjustified)
+- No BoltDB (using dual zarray COW for application-level checkpointing)
+- No application-layer encryption (trust OS boundary: chmod 0700, LUKS/dm-crypt)
+- No WAL logging for job deletion (deferred to checkpoint time to simplify replay)

.kiro/steering/security.md

@@ -0,0 +1,61 @@
+# Security Standards
+
+## Philosophy
+
+Defense in depth, least privilege, fail closed. Authentication verifies *who*; authorization verifies *what they can do*. Centralize both — never duplicate permission checks across handlers.
+
+## RBAC Enforcement
+
+All protected routes use `RequirePermission` middleware attached at registration time:
+
+```go
+// In router/routers.go
+hostsGroup.GET("", RequirePermission("host:view"), handler)
+hostsGroup.POST("/control", RequirePermission("host:control"), handler)
+```
+
+`RequirePermission` in `controllers/rbac_check.go` resolves the user from the JWT, looks up their effective permissions (including inherited role permissions), and aborts with 403 if the required permission is absent.
+
+**cluster_admin** bypasses all permission checks via early return — do not add new bypass paths outside this single mechanism.
+
+## Permission String Format
+
+`"action:scope"` — e.g., `"job:submit"`, `"host:view"`, `"queue:manage"`.
+
+Scopes: `all` (unrestricted), `self` (own resources), `cluster`, `hostgroup`, `self_ug`, `self_hg`.
+
+Permissions are defined in `config/rbac.yaml`. Roles support inheritance via `includes:` — resolution is recursive. Add new permissions to the YAML first, then wire them onto routes.
+
+## JWT Authentication
+
+Token flow:
+1. `POST /auth/logon` → validates credentials via configured auth backend → issues JWT with `{username, IP, expiry}` claims
+2. Every subsequent request: `Authorization: Bearer <token>` header
+3. `middleware/validatehandler.go` strips the prefix, parses with golang-jwt, **validates the IP claim against the request IP**
+
+IP-bound tokens prevent token theft across network boundaries. Do not remove the IP validation.
+
+Token secret is loaded from `config/auth.yaml` at startup. It must never appear in source code or logs.
+
+## Auth Backends
+
+Backend selection is controlled by `config/auth.yaml`. Supported: `configfile`, `ldap`, `oauth2`, `pam`. All backends implement the same interface — adding a new backend means implementing that interface, not modifying middleware.
+
+## Input Validation
+
+Validate at the API boundary inside `preCheck{Resource}Request()` helpers — before any service or store access. Return 400 with a clear message on validation failure. Do not pass unvalidated input to downstream layers.
+
+## Secrets
+
+- JWT secret key: `config/auth.yaml` (file-system protected, chmod 0600)
+- LDAP bind credentials: `config/auth.yaml`
+- **Never log** tokens, passwords, secret keys, or full request bodies containing credentials
+- Validate all required config at startup; fail fast with a clear error if missing
+
+## Rate Limiting and IP Whitelist
+
+Rate limiting (`middleware/rate_limit.go`) and IP whitelist (`middleware/ip_whitelist.go`) are applied before JWT validation in the middleware chain. They use `ctx.ClientIP()` and abort with 429 / 403 respectively. Both are configured in `config/` YAML — do not hardcode thresholds.
+
+## Transport
+
+All daemon-to-daemon communication is over TCP. Production deployments add TLS at the network layer (not handled in application code). The WAL and replication wire format uses Cap'n Proto — schema files in `src/schema/`. Do not add unvalidated deserialization paths for external input.

.kiro/steering/structure.md

@@ -0,0 +1,99 @@
+# Project Structure
+
+## Organization Philosophy
+
+**Layered monorepo, component-per-directory.** Each major subsystem lives under `src/<component>/` with its own Meson build file and CLAUDE.md. The Go frontend is isolated in `src/frontend/` as two independent Go modules. All C daemons share `src/lib/` (libxce) and public headers from `src/include/xce/`.
+
+```
+src/
+├── lib/           C — shared library (libxce): data structures, IPC, utilities
+├── controller/    C — cluster controller daemon
+├── worker/        C — per-node job execution daemon (xcecd)
+├── scheduler/     C — job scheduling engine (plugin architecture)
+├── autoscaler/    C — autoscaling engine (provider plugins)
+├── frontend/      Go — apiserver (Gin REST) + xcecli (Cobra CLI)
+├── include/xce/   C — public shared headers consumed across all daemons
+├── schema/        Cap'n Proto schema files (c.capnp, controller.capnp)
+└── docs/          Project-wide documentation (markdown)
+```
+
+## C Daemon Layout Pattern
+
+Each C daemon (`controller/`, `worker/`, `scheduler/`, `autoscaler/`) follows:
+
+```
+src/<component>/
+    meson.build          — build definition
+    CLAUDE.md            — component-specific AI guide
+    include/             — component-internal headers
+    src/                 — .c implementation files (flat, one domain per file)
+    testsuite/           — test programs (test_*.c)
+```
+
+**Exception**: `controller/` keeps its `.c` files flat in the component root (not in a nested `src/`). All other C daemons use `src/<component>/src/`.
+
+Plugin-based components add their own subdirectory:
+- `scheduler/` — plugins in `src/plugins/`
+- `autoscaler/` — providers in `src/autoscaler/providers/`
+
+## Go Frontend Layout Pattern
+
+**apiserver** (`src/frontend/apiserver/`):
+
+| Package | Purpose |
+|---------|---------|
+| `cmd/` | Entry point, init sequence |
+| `router/` | Route registration + middleware chain |
+| `controllers/` | HTTP handlers — one file per resource |
+| `services/` | Business logic |
+| `store/` | Thread-safe in-memory KV stores (generics) |
+| `models/` | Request/response DTOs, domain structs |
+| `middleware/` | CORS, rate limiting, JWT, IP whitelist |
+| `backend/` | CGO bridge to C library |
+| `backend/auth/` | Pluggable auth providers |
+| `utils/` | Zap logger wrapper, config parsing, crypto |
+| `common/` | Global error codes and response helpers |
+| `config/` | YAML configs + JSON schemas |
+
+## Naming Conventions
+
+### C
+
+- **Files**: `snake_case.c` / `snake_case.h` — one domain per file
+- **Public headers**: `src/include/xce/<name>.h` — the libxce API boundary
+- **Component headers**: `src/<component>/include/<name>.h`
+- **Test files**: `test_<unit>.c` (e.g., `test_ha_cluster.c`, `test_persist_wal.c`)
+- **Functions**: `snake_case` with component prefix — `ha_*`, `cluster_*`, `xcecd_*`, `xce_*`
+- **Types**: `snake_case_t` suffix (e.g., `cluster_t`, `ha_state_t`)
+- **Struct tags**: `struct <name>_s` (e.g., `struct cluster_s`)
+- **Enums/Macros**: `UPPER_SNAKE_CASE` with component prefix (`CTRL_PRIMARY`, `HA_DEFAULT_*`)
+
+### Go
+
+- **Files**: one file per resource — `jobs.go`, `queues.go`, `hosts.go`
+- **Tests**: `*_test.go` co-located; integration tests as `integration_test.go`, `<resource>_control_test.go`
+- **Controller structs**: `{Resource}Controller`
+- **Handler methods**: `{Action}{Resource}` (e.g., `SubmitJob`, `ListJobs`)
+- **Permission strings**: `"action:scope"` (e.g., `"job:view:self"`)
+- **JSON tags**: `snake_case`
+
+## Test Organization
+
+**C tests** (`testsuite/` inside each component):
+- Standalone `.c` programs, one per unit: `test_<unit>.c`
+- `test.c` provides the test runner harness
+- Tiered naming: `test_p1_integration.c`, `test_e2e_wal.c`, `test_p3_hardening.c`
+- Test-only symbols guarded by `#ifdef XCE_TESTING`
+
+**Go tests** (co-located `_test.go`):
+- Table-driven unit tests, `t.TempDir()` for fixtures
+- Global state reset helpers (e.g., `resetRBACState()`) for test isolation
+- `make test` = unit only; `make test-all` = unit + integration
+
+## Code Organization Principles
+
+- **One domain per file** in C — `ha.c` owns HA logic, `job.c` owns job logic. No megafiles.
+- **Single Go module per binary** — apiserver and xcecli share no Go code at the module level.
+- **Controllers delegate to services** — HTTP handlers validate input and delegate; business logic lives in `services/`.
+- **Dependencies flow inward**: `controllers → services → store/backend`, never the reverse.
+- **Cap'n Proto for wire formats** — any new inter-daemon or persistence protocol uses `.capnp` schemas in `src/schema/`.

.kiro/steering/tech.md

@@ -0,0 +1,100 @@
+# Technology Stack
+
+## Architecture
+
+```
+xcecli (Go/Cobra) ──REST──► apiserver (Go/Gin) ──CGO──► C daemon layer
+                                                          controller / scheduler / worker / autoscaler
+                                                          ↑↑↑ all linked against libxce (src/lib/)
+```
+
+Request path inside apiserver: `Gin Router → Middleware (CORS/RateLimit/IPWhitelist/JWT) → RBAC Check → Controller → Services → In-Memory Store`
+
+All controller state is in-memory, seeded at startup from YAML config. No persistent database in the API tier. C daemons communicate via libtirpc RPC; WAL + checkpoint provide persistence at the controller level.
+
+## Core Technologies
+
+- **Go 1.23.4** — apiserver (REST API) and xcecli (CLI). Two independent Go modules, no `go.work`.
+- **C (gnu17)** — controller, worker, scheduler, autoscaler, libxce. No C++.
+- **Gin v1.10.0** — HTTP router, middleware, binding
+- **Cobra v1.9.1** — CLI framework for xcecli
+- **Zap v1.27.0** — structured logging throughout (both Go modules)
+- **Viper v1.19.0 / v1.20.1** — YAML config management
+- **Cap'n Proto** — wire serialization for WAL and daemon-to-daemon replication
+- **libtirpc** — inter-daemon RPC transport
+
+## Build Systems
+
+| System | Scope |
+|--------|-------|
+| **Meson ≥ 1.5** | C daemons + Go modules via `tools/go_build` wrapper |
+| **Make** | Go development loop (build/test) from `src/frontend/` |
+| **Nix flakes** | Hermetic dev environment, AppImage packaging |
+
+```bash
+# Full build (C + Go, debug + ASAN)
+/xce-build
+
+# Go development loop (from src/frontend/)
+make build-server          # → /tmp/apiserver
+make build-cli             # → /tmp/xce_static (CGO_ENABLED=0)
+make test                  # unit tests
+make test-integration      # regression suite (needs XCE_BIN + SRV_URL)
+
+# Nix package builds (from src/)
+nix build .#xcecore        # C daemons + apiserver
+nix build .#xcecore_debug  # debug + ASAN
+nix build .#xcecli         # CLI only
+```
+
+## Development Standards
+
+### Controller Pattern (Go)
+
+```go
+// Struct: {Resource}Controller
+type JobController struct { ... }
+
+// Handler: {Action}{Resource}
+func (c JobController) SubmitJob(ctx *gin.Context) {
+    // Validate input
+    req, err := preCheckSubmitJobRequest(ctx)
+    if err != nil { ReturnError(ctx, code, err, nil); return }
+
+    // Business logic
+    result, err := c.svc.Submit(req)
+    if err != nil { ReturnError(ctx, code, err, nil); return }
+
+    ReturnSuccess(ctx, code, "submitted", result)
+}
+```
+
+Early return on every error. Log with zap fields before returning.
+
+### Testing Requirements
+
+- **TDD mandatory**: Write failing test first, then implementation.
+- **Go unit tests**: Table-driven, no external dependencies, `t.TempDir()` for fixtures.
+- **C tests**: Add test case in `testsuite/` before modifying implementation.
+- `make test` must pass without a live server.
+- `make test-integration` requires `XCE_BIN` + `SRV_URL` env vars.
+
+### Code Formatting
+
+- Go: `gofmt` / `goimports` (post-edit hooks configured)
+- C: `clang-format` using project config at `xce/.clang-format`
+
+### Auth and RBAC
+
+- Auth backend set in `config/auth.yaml`. Supports: configfile, LDAP, OAuth2, PAM.
+- Permissions as `"action:scope"` strings: `"job:view:self"`, `"queue:submit:all"`.
+- RBAC loaded from `config/rbac.yaml`, enforced by `controllers.RequirePermission()` middleware.
+
+## Key Technical Decisions
+
+1. **No Go workspace** — `apiserver` and `xcecli` are independent modules; changes to shared types require updating both.
+2. **In-memory API state** — no database in Go tier; state loaded from YAML at startup via `store.LoadBuiltinData()`.
+3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell.
+4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin.
+5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output.
+6. **Single-writer HA** — controller is always one active primary; standby election via LSF-style priority, not consensus.

.kiro/steering/testing.md

@@ -0,0 +1,89 @@
+# Testing Standards
+
+## Philosophy
+
+- **TDD mandatory**: Write a failing test first, then the minimal implementation. The cycle is Red → Green → Refactor.
+- Test behavior, not implementation details.
+- Tests must run without a live server or external services.
+- A change is not complete until `make test` (and `make test-integration` where applicable) passes.
+
+## Go Tests (apiserver / xcecli)
+
+**Co-located** alongside source: `jobs.go` → `jobs_test.go`. Integration tests use descriptive names: `integration_test.go`, `host_control_test.go`.
+
+**Table-driven** for validation/filter logic:
+
+```go
+tests := []struct {
+    name     string
+    input    SomeRequest
+    wantErr  bool
+}{
+    {"valid request", validReq, false},
+    {"missing field", missingReq, true},
+}
+for _, tt := range tests {
+    t.Run(tt.name, func(t *testing.T) {
+        got, err := preCheckRequest(tt.input)
+        if (err != nil) != tt.wantErr { t.Errorf(...) }
+    })
+}
+```
+
+**State isolation** — reset global store state between tests:
+
+```go
+func resetRBACState() { /* clear in-memory stores */ }
+
+func TestPermissions(t *testing.T) {
+    t.Cleanup(resetRBACState)
+    // ...
+}
+```
+
+**Fixtures**: use `t.TempDir()` for file-system state; never rely on absolute paths or external services.
+
+**Build targets**:
+- `make test` — unit tests only (`CGO_ENABLED=0`)
+- `make test-backend` — CGO-backed backend tests (requires nix dev shell + build dir)
+- `make test-integration` — full regression suite (requires `XCE_BIN` and `SRV_URL` env vars)
+
+## C Tests (controller / worker / scheduler / lib)
+
+**Location**: `testsuite/` directory inside each C component.
+
+**Structure**: one `.c` test program per unit:
+
+```
+src/controller/testsuite/
+    test.c                       ← test runner harness
+    test_ha_cluster.c            ← tests for ha.c
+    test_persist_wal.c           ← WAL-specific tests
+    test_e2e_checkpoint.c        ← end-to-end checkpoint flow
+    test_p1_integration.c        ← P1 HA integration tier
+```
+
+File naming convention: `test_<unit>.c`. E2E tests: `test_e2e_<topic>.c`. Integration tiers: `test_p1_*.c`, `test_p2_*.c`, `test_p3_hardening.c`.
+
+**Test-only symbols** are guarded to prevent production bloat:
+
+```c
+#ifdef XCE_TESTING
+extern void reset_ha_state(ha_state_t *s);
+#endif
+```
+
+**Add a test case in `testsuite/` before modifying any C implementation.**
+
+## Coverage Targets
+
+- Go: 80% minimum on business logic (controllers, services, store)
+- C: every new function in a daemon must have at least one test case in `testsuite/`
+- Integration: all public REST endpoints must have at least one integration test case in `regression/`
+
+## What Not to Mock
+
+- Mock **external dependencies only** (network calls, file I/O that crosses component boundaries)
+- Do **not** mock the system under test
+- In-memory stores (`store/`) are real — use them directly in unit tests
+- CGO bridge (`backend/`) is covered by `make test-backend` — do not stub it in unit tests