chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x), heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup. New: ha-consistency-invariants umbrella (requirements/design/research) and its four implementation sub-specs (specs generated via batch, cross-spec review applied), steering roadmap + product.md Raft+CP decision.
This commit is contained in:
parent
084244941b
commit
06135954ff
88 changed files with 25611 additions and 5285 deletions
99
CLAUDE.md
99
CLAUDE.md
|
|
@ -7,80 +7,33 @@ This project uses OpenWolf for context management. Read and follow .wolf/OPENWOL
|
|||
|
||||
# XCE Cluster Management System
|
||||
|
||||
## Overview
|
||||
XCE is an HPC cluster management system providing job scheduling, queue management, and cluster resource control. The Go-based API/CLI tier fronts C system daemons (controller, worker, scheduler).
|
||||
## Project Overview
|
||||
XCE is an HPC cluster management system — Go API/CLI tier fronting C daemons (controller, worker, scheduler). Full context in `.kiro/steering/product.md`.
|
||||
|
||||
## Tech Stack
|
||||
- **Go 1.23.4**: REST API server (`src/frontend/apiserver`), CLI tool (`src/frontend/xcecli`)
|
||||
- **C (gnu17)**: Core daemons — controller, worker, scheduler, shared lib
|
||||
- **Build**: Meson + Nix flakes for C components; `make` for Go components
|
||||
- **Go key deps**: Gin v1.10.0, JWT v4, Zap logging, Viper config, Cobra CLI
|
||||
## Engineering Guide
|
||||
|
||||
## Development Commands
|
||||
```bash
|
||||
# Build with /xce-build skill (debug + ASAN, clean build, runs tests)
|
||||
/xce-build # Build all components + run all tests
|
||||
/xce-build lib # Build libxce + run lib tests
|
||||
/xce-build controller # Build lib + controller + run controller tests
|
||||
/xce-build scheduler # Build lib + scheduler + run scheduler tests
|
||||
/xce-build worker # Build lib + worker (no unit tests registered)
|
||||
/xce-build apiserver # Build lib + apiserver + xcecli + run Go unit tests
|
||||
/xce-build xcecli # Build xcecli only + run xcecli Go tests
|
||||
For details on any topic below, read the corresponding steering file in `.kiro/steering/`. CLAUDE.md provides pointers, not duplicates.
|
||||
|
||||
```
|
||||
| Topic | Steering File |
|
||||
|-------|---------------|
|
||||
| Project purpose, HA phases, design decisions | `product.md` |
|
||||
| Technology stack, build systems, key decisions | `tech.md` |
|
||||
| Directory layout, naming conventions, test org | `structure.md` |
|
||||
| Build commands, Nix flake, deploy, AppImage | `build-deployment.md` |
|
||||
| REST handler pattern, daemon opcodes, Cap'n Proto | `api-standards.md` |
|
||||
| HA protocol, async I/O, Raft election, heartbeat | `ha-async-io.md` |
|
||||
| TDD rules, C + Go test patterns, coverage targets | `testing.md` |
|
||||
| Error handling: Go ReturnError, C errnox, zap logging | `error-handling.md` |
|
||||
| RBAC, JWT, auth backends, input validation, secrets | `security.md` |
|
||||
|
||||
## Test-Driven Development
|
||||
Follow TDD strictly throughout the full development process:
|
||||
## Key Rules (from steering — summarized for quick reference)
|
||||
|
||||
1. **Red** — Write a failing test that specifies the desired behaviour before writing any implementation code.
|
||||
2. **Green** — Write the minimal implementation to make the test pass. Do not over-engineer.
|
||||
3. **Refactor** — Clean up the implementation and tests while keeping all tests green.
|
||||
|
||||
### Rules
|
||||
- Never implement a feature without a corresponding test written first.
|
||||
- Tests must be runnable without a live server or external services (use table-driven tests, stubs, and `t.TempDir()` for fixtures).
|
||||
- A change is not complete until `make test` (and `make test-integration` where applicable) passes.
|
||||
- For C components, add a test case in the relevant `testsuite/` before modifying the implementation.
|
||||
- Tests live next to the code they cover (`_test.go` for Go, `testsuite/` for C).
|
||||
- Keep test scope narrow — unit-test business logic; use integration tests for HTTP/API surface.
|
||||
|
||||
## Architecture
|
||||
```
|
||||
Request → Gin Router → Middleware (CORS/RateLimit/IPWhitelist/JWT) → RBAC Check → Controller → Services → In-Memory Store
|
||||
```
|
||||
- **3 independent Go modules**: `apiserver`, `xcecli`, `bcctool` (no go.work)
|
||||
- **Stateful API server**: in-memory stores seeded at startup via `store.LoadBuiltinData()`
|
||||
- **Pluggable auth**: configfile / LDAP / OAuth2 / PAM (set in `config/auth.yaml`)
|
||||
- **RBAC**: domain-based role-action-domain model loaded from `config/rbac.yaml`
|
||||
|
||||
## Connection Model and Async I/O
|
||||
|
||||
### Long-Lived Persistent Connections (CRITICAL)
|
||||
All XCE daemon connections are **long-lived** — they are established once and reused for the lifetime of the role relationship:
|
||||
|
||||
| Connection | Direction | Owner | Purpose |
|
||||
|---|---|---|---|
|
||||
| Controller primary → standby | Primary dials out during election | Primary (`ha->standby_endpoints`) | Config push, WAL replication, heartbeat |
|
||||
| Controller standby ← primary | Inbound on standby listener | Standby (`primary_ctrl_fd`) | Receives ctrlhello, WAL, config, heartbeat |
|
||||
| Primary → worker | Primary dials out | Primary (`c->workers[i].ep`) | Job dispatch, heartbeat |
|
||||
| Scheduler → primary | Scheduler dials out | Scheduler | Schedule requests |
|
||||
| API server → primary | API server dials out | API server backend | Job submit, cluster queries |
|
||||
|
||||
Connections are **never** closed and re-opened between RPCs. Closing a connection signals a failure event (RDHUP) which triggers failover or reconnect logic.
|
||||
|
||||
### Async Non-Blocking I/O Model (CRITICAL)
|
||||
All C daemon I/O runs through **iomgr** (epoll-based) + **vtaskmgr** (thread pool):
|
||||
|
||||
- **iomgr** — single-threaded epoll loop; dispatches inbound packets as vtasks; manages endpoint lifetime
|
||||
- **vtaskmgr** — thread-pool executing vtask handlers; handlers must return quickly (no blocking I/O)
|
||||
- **delaymgr** — timer thread for periodic/delayed tasks; the only place blocking operations (connect, file I/O) are permitted
|
||||
- `xce_request(ep, &req)` — enqueues a packet to the endpoint's send buffer; **always non-blocking**
|
||||
- `xce_reply(ep, host, &reply)` — enqueues a reply; **always non-blocking**
|
||||
|
||||
**Never block in a vtask handler.** Blocking calls (connect, read, usleep) belong in delaymgr tasks or dedicated threads.
|
||||
|
||||
### Primary Heartbeat to Standbys
|
||||
The primary sends `OPCode_nop` to each standby every `heartbeat_interval_ms` via a self-perpetuating delaymgr one-shot task (`ha_start_primary_heartbeat`). The task re-arms itself only while role is PRIMARY or PRIMARY_COMMITTING — on demotion the loop stops automatically (no cancel API needed). This keeps the standby-side watchdog (`ha_arm_heartbeat_timeout_vtask`) from timing out during idle periods.
|
||||
- **TDD mandatory**: red → green → refactor; test in `testsuite/` before C changes, `_test.go` before Go changes
|
||||
- **Async I/O**: never block in vtask handlers; delaymgr for blocking ops. `xce_request`/`xce_reply` are always non-blocking
|
||||
- **Connections are long-lived**: closing signals RDHUP failover. Inbound connections are vote-only; outbound carry all state push
|
||||
- **Middleware chain** (non-negotiable): WhitelistURL → WhitelistIP → RateLimit → JWT → CORS → RBAC
|
||||
- **Error pattern**: `ReturnError(code, msg, nil); return` — early return on every error
|
||||
- **Build**: `/xce-build [component]` — debug + ASAN + tests. See `build-deployment.md` for full commands
|
||||
|
||||
## Key Files
|
||||
- `src/frontend/Makefile` — primary build/test orchestration
|
||||
|
|
@ -145,7 +98,7 @@ Kiro-style Spec-Driven Development on an agentic SDLC
|
|||
- Follow the user's instructions precisely, and within that scope act autonomously: gather the necessary context and complete the requested work end-to-end in this run, asking questions only when essential information is missing or the instructions are critically ambiguous.
|
||||
|
||||
## Steering Configuration
|
||||
- Load entire `.kiro/steering/` as project memory
|
||||
- Core files: `product.md`, `tech.md`, `structure.md`
|
||||
- XCE custom files: `testing.md`, `error-handling.md`, `security.md`
|
||||
- Add new custom files via `/kiro-steering-custom`
|
||||
- Load entire `.kiro/steering/` as project memory — all `.md` files are authoritative
|
||||
- Core: `product.md`, `tech.md`, `structure.md`
|
||||
- Custom: `testing.md`, `error-handling.md`, `security.md`, `ha-async-io.md`, `api-standards.md`, `build-deployment.md`
|
||||
- Add new custom files via `/kiro-steering-custom`; sync existing via `/kiro-steering`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue