chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x), heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup. New: ha-consistency-invariants umbrella (requirements/design/research) and its four implementation sub-specs (specs generated via batch, cross-spec review applied), steering roadmap + product.md Raft+CP decision.
This commit is contained in:
parent
084244941b
commit
06135954ff
88 changed files with 25611 additions and 5285 deletions
144
.kiro/steering/api-standards.md
Normal file
144
.kiro/steering/api-standards.md
Normal file
|
|
@ -0,0 +1,144 @@
|
|||
# API Standards
|
||||
|
||||
## Philosophy
|
||||
|
||||
Two API surfaces: Go REST API (apiserver) and C daemon messaging (Cap'n Proto over TCP). Both follow a consistent pattern: validate at the boundary, delegate business logic, return structured responses. Non-blocking I/O throughout the daemon layer.
|
||||
|
||||
## Go REST API (apiserver)
|
||||
|
||||
### Handler Pattern
|
||||
|
||||
Every resource follows a single consistent pattern:
|
||||
|
||||
```go
|
||||
// {Resource}Controller struct
|
||||
type JobController struct{}
|
||||
|
||||
// {Action}{Resource} handler method
|
||||
func (j JobController) SubmitJob(ctx *gin.Context) {
|
||||
// 1. Validate input at the boundary
|
||||
req, err := preCheckSubmitJobRequest(ctx)
|
||||
if err != nil {
|
||||
ReturnError(ctx, common.INVALID_PARAMS_CODE, err.Error(), nil)
|
||||
return
|
||||
}
|
||||
|
||||
// 2. Delegate to service layer
|
||||
result, err := j.svc.Submit(req)
|
||||
if err != nil {
|
||||
ReturnError(ctx, common.ERROR_CODE, err.Error(), nil)
|
||||
return
|
||||
}
|
||||
|
||||
// 3. Return structured success
|
||||
ReturnSuccess(ctx, common.SUCCESS_CODE, "submitted", result)
|
||||
}
|
||||
```
|
||||
|
||||
**Rules**:
|
||||
- Early return on every error — `ReturnError(...); return`
|
||||
- Keep `preCheck*` functions in the same file as the handler
|
||||
- Controllers delegate to services — business logic never lives in handlers
|
||||
- Dependencies flow inward: `controllers → services → store/backend`
|
||||
|
||||
### Response Format
|
||||
|
||||
Every HTTP response uses a consistent envelope via two helpers in `controllers/response.go`:
|
||||
|
||||
```go
|
||||
// Success (always HTTP 200)
|
||||
ReturnSuccess(ctx, code, msg, data)
|
||||
// → {code: 0, msg: "success", data: {...}}
|
||||
|
||||
// Error (HTTP status == code parameter)
|
||||
ReturnError(ctx, code, msg, details)
|
||||
// → {code: 401, msg: "access forbidden", details: nil}
|
||||
```
|
||||
|
||||
### Standard Error Codes
|
||||
|
||||
All codes defined in `common/common.go`:
|
||||
|
||||
| Code | Constant | Usage |
|
||||
|------|----------|-------|
|
||||
| 0 | `SUCCESS_CODE` | All success responses |
|
||||
| 400 | `INVALID_PARAMS_CODE` | Malformed/missing input |
|
||||
| 401 | `ERROR_AUTH_CHECK_TOKEN_CODE` | Missing/invalid JWT |
|
||||
| 403 | `ERROR_AUTH_CODE` | Valid JWT, insufficient permission |
|
||||
| 404 | `ERROR_NOT_FOUND_CODE` | Resource not found |
|
||||
| 429 | `ERROR_TOO_MANY_REQUESTS_CODE` | Rate limit exceeded |
|
||||
| 500 | `ERROR_CODE` | Unexpected server error |
|
||||
|
||||
### Middleware Chain
|
||||
|
||||
Order is defined in `router/routers.go` and is **non-negotiable**:
|
||||
|
||||
```
|
||||
WhitelistURL → WhitelistIP → RateLimit → ValidateHandler (JWT) → CORS → RBAC
|
||||
```
|
||||
|
||||
- **WhitelistURL**: filter by allowed paths (applied first for efficiency)
|
||||
- **WhitelistIP**: abort 403 if client IP not in whitelist
|
||||
- **RateLimit**: 10 req/sec, burst 20 (global config)
|
||||
- **ValidateHandler**: parse and validate JWT from `Authorization: Bearer` header; validate body content-type
|
||||
- **CORS**: origin checks via config
|
||||
- **RBAC**: `RequirePermission("action:scope")` per-route, enforced by `controllers/rbac_check.go`
|
||||
|
||||
### Permission Format
|
||||
|
||||
`"action:scope"` — scopes: `all`, `self`, `cluster`, `hostgroup`, `self_ug`, `self_hg`.
|
||||
Defined in `config/rbac.yaml`. Roles support inheritance via `includes:`.
|
||||
|
||||
## C Daemon Messaging
|
||||
|
||||
### Cap'n Proto Wire Format
|
||||
|
||||
All inter-daemon messages are Cap'n Proto serialized. Schema: `src/schema/controller.capnp`.
|
||||
|
||||
**Request** (`xce_request_t`):
|
||||
- `opcode` — OPCode enum (38 values)
|
||||
- `seq` — request sequence number
|
||||
- `from` — sender peer info (host/key)
|
||||
- `body` — union of 27 message types
|
||||
|
||||
**Reply** (`xce_reply_t`):
|
||||
- `opcode` — always `OPCode_reply` or `OPCode_redirect`
|
||||
- `status` — error code (0 = success)
|
||||
- `seq` — echoes request seq
|
||||
- `data` — union of reply payloads (clusterconf, jobid, stateReport, walackreplymsg, etc.)
|
||||
|
||||
### Key OPCodes
|
||||
|
||||
| Group | OPCodes | Purpose |
|
||||
|-------|---------|---------|
|
||||
| Control | `jobsubmit (11)`, `jobdispatch (8)`, `jobcontrol (15)` | Job lifecycle |
|
||||
| Cluster | `hello (2)`, `join (3)`, `clusterconf (17)`, `heartbeat (25)` | Membership and liveness |
|
||||
| Replication | `replConfig (29)`, `replConfigPush (30)`, `walCatchupEntry (37)`, `snapshotTransfer (36)` | Config + WAL replication |
|
||||
| Vote | `requestVote`, reply | Raft election |
|
||||
| Data | `push (6)`, `get (5)`, `notify (10)` | Data exchange |
|
||||
|
||||
### Daemon I/O Functions
|
||||
|
||||
```c
|
||||
// Enqueue request to endpoint send buffer (non-blocking, always)
|
||||
int xce_request(endpoint_t *ep, xce_request_t *req);
|
||||
|
||||
// Enqueue reply to inbound request sender (non-blocking)
|
||||
int xce_reply(endpoint_t *ep, xce_peerinfo_t *host, xce_reply_t *reply);
|
||||
|
||||
// Cap'n Proto serialize request to packet
|
||||
int xce_encode(xce_request_t *req, io_packet_t *pkt);
|
||||
|
||||
// Cap'n Proto deserialize packet to request
|
||||
int xce_decode(io_packet_t *pkt, xce_request_t *req);
|
||||
```
|
||||
|
||||
**msg_handler callback**: Each endpoint has `ep->msg_handler` — a function pointer `int (*)(void*)` called by iomgr when a packet arrives. Handlers must return quickly (vtask context). Registered per-endpoint during connection setup.
|
||||
|
||||
### Adding a New Opcode
|
||||
|
||||
1. Add the opcode constant to `src/include/request.h` (OPCode enum)
|
||||
2. Add the message struct to `src/schema/controller.capnp` (Request/Reply union arms)
|
||||
3. Rebuild capnp schemas (automatic via meson)
|
||||
4. Wire the handler in the dispatch gate (`ha_dispatch_gate` in `ha.c`)
|
||||
5. Add encode/decode paths in `src/lib/request.c` if using raw payloads
|
||||
137
.kiro/steering/build-deployment.md
Normal file
137
.kiro/steering/build-deployment.md
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
# Build & Deployment
|
||||
|
||||
## Philosophy
|
||||
|
||||
A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Every build includes debug symbols + ASAN. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management.
|
||||
|
||||
## Build Commands
|
||||
|
||||
All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
|
||||
|
||||
```bash
|
||||
/xce-build # all components + all tests
|
||||
/xce-build lib # libxce + lib tests
|
||||
/xce-build controller # lib + controller + controller tests
|
||||
/xce-build scheduler # lib + scheduler + scheduler tests
|
||||
/xce-build worker # lib + worker (daemon only, no unit tests)
|
||||
/xce-build apiserver # lib + apiserver + xcecli + Go unit tests
|
||||
/xce-build xcecli # xcecli + Go tests
|
||||
```
|
||||
|
||||
Each invocation:
|
||||
1. `nix develop` into the hermetic shell
|
||||
2. `meson setup build --buildtype=debug -Db_sanitize=address` (fresh clean build)
|
||||
3. `ninja -C build` (compile all + test executables)
|
||||
4. Run registered meson tests
|
||||
|
||||
## ASAN Integration
|
||||
|
||||
ASAN is enabled in every build by default (`-Db_sanitize=address`). Controller tests set `detect_leaks=0` (known pre-existing leaks flagged as technical debt). Test binaries print LeakSanitizer reports on stderr after exit.
|
||||
|
||||
## Meson Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── meson.build # top-level: subdir() order, project()
|
||||
├── lib/meson.build # libxce shared library + libxce_test
|
||||
├── controller/meson.build # xce_controller daemon + controller_unit_test
|
||||
├── worker/meson.build # xcecd daemon
|
||||
├── scheduler/meson.build # scheduler daemon + test
|
||||
├── autoscaler/meson.build # autoscaler daemon
|
||||
├── frontend/apiserver/meson.build # Go apiserver via tools/go_build
|
||||
└── frontend/xcecli/meson.build # Go xcecli via tools/go_build
|
||||
```
|
||||
|
||||
**Dependency flow**: `lib → {controller, worker, scheduler, apiserver}`
|
||||
|
||||
**Test registration**:
|
||||
```python
|
||||
# controller/meson.build
|
||||
test('controller',
|
||||
controller_unit_test,
|
||||
suite: 'controller',
|
||||
timeout: 300,
|
||||
env: ['ASAN_OPTIONS=detect_leaks=0'])
|
||||
```
|
||||
|
||||
**XCE_TESTING flag**: The controller_unit_test executable compiles with `-DXCE_TESTING` to enable test-only seams (hook variables, reset functions).
|
||||
|
||||
## Nix Flake
|
||||
|
||||
Located at `src/flake.nix`. Provides:
|
||||
|
||||
```bash
|
||||
nix develop # hermetic dev shell (meson, pkgconfig, capnp, packcc, Go 1.23.4, etc.)
|
||||
nix build .#xcecore # C daemons + apiserver (release)
|
||||
nix build .#xcecore_debug # default package: debug + ASAN
|
||||
nix build .#xcecli # CLI only
|
||||
```
|
||||
|
||||
**AppImage**: The flake produces an AppImage wrapping `xce` (shell entry point) with bundled daemons, haproxy, glow, and less. The entry script reads from `/tools/appimage/entry.sh`.
|
||||
|
||||
## Local HA Cluster Deployment
|
||||
|
||||
`/xce-deploy-ha` (skill at `.claude/skills/xce-deploy-ha/`):
|
||||
|
||||
```bash
|
||||
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
|
||||
```
|
||||
|
||||
This creates under `/tmp/xce-deploy/`:
|
||||
```
|
||||
certs/ # TLS certs for all peers
|
||||
controller/
|
||||
ctrl-1/
|
||||
conf/{current, cluster.bin} # primary's config
|
||||
data/ # WAL + checkpoint storage
|
||||
ctrl-2/
|
||||
conf/ # receives config via OPCode_replConfigPush
|
||||
data/
|
||||
ctrl-3/ (same as ctrl-2)
|
||||
logs/ # per-node log files
|
||||
scripts/
|
||||
start.sh ctrl-N # launch a single node
|
||||
cleanup.sh # wipe data/standby-conf/logs, keep ctrl-1 config + certs
|
||||
```
|
||||
|
||||
**Startup** (from deploy dir):
|
||||
```bash
|
||||
VERBOSE=-vvv scripts/start.sh ctrl-1 & # primary first
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv scripts/start.sh ctrl-2 & # standby (0.5s stagger for election ordering)
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv scripts/start.sh ctrl-3 &
|
||||
```
|
||||
|
||||
**Clean restart** (keeps certs + ctrl-1 config, wipes all data):
|
||||
```bash
|
||||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
bash scripts/cleanup.sh
|
||||
# Then re-start all nodes
|
||||
```
|
||||
|
||||
## Go Frontend Build
|
||||
|
||||
Go builds use `src/frontend/Makefile` for development, Meson for production:
|
||||
|
||||
```bash
|
||||
make build-server # → /tmp/apiserver
|
||||
make build-cli # → /tmp/xce_static (CGO_ENABLED=0)
|
||||
make test # unit tests (CGO_ENABLED=0)
|
||||
make test-backend # CGO-backed backend tests (needs nix dev shell)
|
||||
make test-integration # full regression suite (needs XCE_BIN + SRV_URL)
|
||||
```
|
||||
|
||||
Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` files, no `go.work`.
|
||||
|
||||
## Production Artifacts
|
||||
|
||||
| Artifact | Build command | Output |
|
||||
|----------|-------------|--------|
|
||||
| Controller daemon | `nix build .#xcecore` | `xce_controller` binary |
|
||||
| Worker daemon | same | `xcecd` binary |
|
||||
| Scheduler daemon | same | `xce_scheduler` binary |
|
||||
| REST API server | same | `apiserver` binary |
|
||||
| CLI tool | `nix build .#xcecli` | `xcecli` binary |
|
||||
| AppImage | `nix build .#xcecore` | `xce-x86_64.AppImage` |
|
||||
| Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries |
|
||||
83
.kiro/steering/ha-async-io.md
Normal file
83
.kiro/steering/ha-async-io.md
Normal file
|
|
@ -0,0 +1,83 @@
|
|||
# HA Protocol & Async I/O
|
||||
|
||||
## Philosophy
|
||||
|
||||
The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O.
|
||||
|
||||
**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).
|
||||
|
||||
## Async I/O Model
|
||||
|
||||
Three components in `src/lib/`:
|
||||
|
||||
| Component | Thread model | Purpose | Blocking? |
|
||||
|-----------|-------------|---------|-----------|
|
||||
| **iomgr** | Single-threaded epoll loop | Inbound packet dispatch, endpoint lifecycle | **Never block** |
|
||||
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | Executes vtask handlers for dispatched packets | **Return quickly** |
|
||||
| **delaymgr** | Single timer thread | Periodic/delayed callbacks, connect(), file I/O | **Permitted** |
|
||||
|
||||
**Critical rule**: Never block in a vtask handler. Blocking calls (connect, read, usleep) belong in delaymgr tasks or dedicated threads.
|
||||
|
||||
Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.
|
||||
|
||||
## Connection Ownership
|
||||
|
||||
**Inbound vote connections** (accepted on the listener): Used **exclusively** for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).
|
||||
|
||||
**Outbound control channels** (`ha->standby_endpoints`): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by `vote_ep_err_handler` (promoted vote endpoints) and `standby_ep_err_handler_primary` (late-joiner outbound connections). Freed on disconnect.
|
||||
|
||||
**Late joiners**: `primary_heartbeat_fn` scans the peers list and opens a fresh outbound connection to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello, registers the endpoint, and calls `repl_add_standby`.
|
||||
|
||||
## Raft Election
|
||||
|
||||
`src/controller/raft.c` implements standard Raft-style election:
|
||||
|
||||
- **RequestVote + RequestVoteReply** over Cap'n Proto
|
||||
- **Freshness gate**: candidate must have `lastLogTerm >= voter` AND `lastLogIndex >= voter`
|
||||
- **Votes persisted atomically** with term increment (raft_cb_persist)
|
||||
- **Majority wins**: quorum = ceil(n_peers / 2)
|
||||
- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning
|
||||
- **WAL truncation** on promotion: `persist_truncate_wal_above(safe_committed_lsn)` removes diverging entries above the safe commit point
|
||||
|
||||
**Key safety invariants**:
|
||||
- `currentTerm` is monotonic (never decreases)
|
||||
- `votedFor` updated atomically with term increment before granting any vote
|
||||
- `commitIndex` never decreases (monotonic)
|
||||
|
||||
## Heartbeat Model
|
||||
|
||||
Bidirectional heartbeat with Raft-style AppendEntries semantics:
|
||||
|
||||
- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY or PRIMARY_COMMITTING.
|
||||
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
|
||||
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
|
||||
|
||||
## Replication Consistency (Two-Tier)
|
||||
|
||||
**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. `CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY` promotion gates on majority convergence (n_peers/2 converged standbys). Single-node mode returns -ENODEV (degraded).
|
||||
|
||||
**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.
|
||||
|
||||
## Dispatch Gate
|
||||
|
||||
`ha_dispatch_gate` controls which opcodes are accepted per role:
|
||||
|
||||
| Role | Allowed |
|
||||
|------|---------|
|
||||
| **CTRL_PRIMARY** | All opcodes |
|
||||
| **CTRL_PRIMARY_COMMITTING** | Replication only (heartbeat, replConfig, walAck, hello); job/queue ops rejected |
|
||||
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes |
|
||||
| **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer |
|
||||
|
||||
This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.
|
||||
|
||||
## Raft-to-Code Mapping
|
||||
|
||||
| TLA+ Variable | XCE Code |
|
||||
|--------------|----------|
|
||||
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) |
|
||||
| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY_COMMITTING, PRIMARY, STANDBY) |
|
||||
| `votedFor[i]` | `ha->voted_for_name[]` |
|
||||
| `log[i]` | WAL segments in `persist->wal_path` |
|
||||
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) |
|
||||
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) |
|
||||
|
|
@ -21,7 +21,7 @@ XCE is an HPC (High-Performance Computing) cluster management system providing j
|
|||
|
||||
XCE provides a **fully integrated single-vendor stack** (no database setup, unified build system) with an open, buildable codebase (Nix flakes, AppImage packaging). Compared to Slurm/PBS, it eliminates the external database requirement. Compared to commercial LSF, it is open source and reproducibly buildable.
|
||||
|
||||
**Key architectural trade-off**: single-writer model throughout — no multi-master consensus. HA failover uses LSF-style priority election (simple, predictable) rather than Raft (10× complexity). All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
|
||||
**Key architectural trade-off**: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via `raft.c`/`raft.h`, and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, `committed_lsn` ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
|
||||
|
||||
## Active Development Context
|
||||
|
||||
|
|
@ -29,15 +29,24 @@ The project is in a **persistence and HA transformation** (branch `persistent_ha
|
|||
|
||||
| Phase | Goal | Status |
|
||||
|-------|------|--------|
|
||||
| P0 | Restart survival (WAL + checkpoint) | Implementing |
|
||||
| P1 | Standby election and takeover | Designed |
|
||||
| P2 | Live replication primary → standby | Designed |
|
||||
| P2.5 | Worker/scheduler auto-reconnect on failover | Designed |
|
||||
| P3–P4 | Hardening, metrics, docs | Planned |
|
||||
| P0 | Restart survival (WAL + checkpoint) | Complete |
|
||||
| P1 | Standby election and takeover (Raft-style) | Complete |
|
||||
| P2 | Live replication primary → standby (config + WAL) | Complete |
|
||||
| P2.5 | Worker/scheduler auto-reconnect on failover | In progress |
|
||||
| P3–P4 | Hardening, metrics, docs | In progress |
|
||||
|
||||
## Spec-Driven Development
|
||||
|
||||
Features are designed and implemented via the kiro spec system (`.kiro/specs/`).
|
||||
Each feature follows a 3-phase approval workflow: Requirements → Design → Tasks → Implementation.
|
||||
Active specs (10 total) include: ha-config-bootstrap, ha-heartbeat-liveness,
|
||||
ha-peer-connection-reuse, ha-repl-hardening, ha-wal-replication, raft-library,
|
||||
election-iomgr-refactor, scheduler-controller-msg-gap, worker-join-refactor,
|
||||
resval-id-decoupling.
|
||||
|
||||
Key design decisions already locked:
|
||||
- No `fork()` for snapshots (mutex inheritance deadlock in multithreaded controller)
|
||||
- No Raft (single writer makes consensus overhead unjustified)
|
||||
- Raft + CP adopted (2026-06-10, supersedes earlier "No Raft"): Raft is the consensus model for both election and log commitment — majority-ACK commit barrier on all state-changing writes, reject (never degrade) when quorum is unavailable. Single-writer log retained as XCE's simplification: the controller is the sole log writer, avoiding concurrent-proposal complexity while inheriting Raft's safety rules (see `.kiro/specs/ha-consistency-invariants/brief.md`)
|
||||
- No BoltDB (using dual zarray COW for application-level checkpointing)
|
||||
- No application-layer encryption (trust OS boundary: chmod 0700, LUKS/dm-crypt)
|
||||
- No WAL logging for job deletion (deferred to checkpoint time to simplify replay)
|
||||
|
|
|
|||
48
.kiro/steering/roadmap.md
Normal file
48
.kiro/steering/roadmap.md
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
# Roadmap
|
||||
|
||||
## Overview
|
||||
|
||||
XCE is in the persistence + HA transformation (branch `persistent_ha`). Governing model: Raft consensus + CP (see `product.md` key decisions). Active spec work is tracked per-spec in `.kiro/specs/`; this file is the **single consolidated future-work document** — the forward-looking queue, long-deferred items, and hardening backlog. It absorbs the former `src/docs/FUTURE_WORK.md` (2026-05-12, config-replication-design review), which now redirects here.
|
||||
|
||||
## In Flight
|
||||
|
||||
- **ha-consistency-invariants** — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.
|
||||
- ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs `/xce-deploy-ha` cluster); sub-spec 4's E2E extends it
|
||||
|
||||
## Specs (dependency order)
|
||||
|
||||
> `[x]` = spec generated (requirements/design/tasks approved, ready for implementation). Implementation tracked via each spec's tasks.md; implement in roadmap order (cross-spec review 2026-06-11: four specs touch ha_dispatch_gate / raft_cb_become_follower in disjoint-but-adjacent regions).
|
||||
|
||||
- [x] ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11: 7 reqs / 4 components / 11 sub-tasks)
|
||||
- [x] ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11: 5 reqs / 6 components / 16 sub-tasks; verified classification supersedes umbrella table for STOP/RESUME/UPDATE/FAIL)
|
||||
- [x] ha-single-leader-role — promotion no-op + own-term gate, CTRL_PRIMARY_COMMITTING removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
|
||||
- [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
|
||||
|
||||
## Future Specs / Backlog (dependency order)
|
||||
|
||||
- [ ] worker-rejoin-resync — **planned long ago, no effort yet (recorded 2026-06-10).** When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): `xce_join_req` carries only `hostattrs_t` (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the `mark_job_*` idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. **Elevated 2026-06-11**: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only).
|
||||
- [ ] admin-config-tool — drives `OP_upgradeConfig` (prerequisite for config-upgrade Phase 2). Encode ClusterConfig YAML/JSON → Cap'n Proto binary, SHA-256 over the bytes, connect to primary (SSL, same CA), send `OP_upgradeConfig`, display reply (version tag, acked count, errors), poll `OP_probe` after restart. Suggested CLI: `xce-config push --controller <host:port> --config cluster.yaml`. MUST validate that user configs don't use the reserved name `"lost_and_found"` for any queue/host (defence-in-depth: controller enforces at runtime via `load_primary_config()` FATAL, tool prevents at authoring time; migration: pre-existing clusters must rename before upgrade — see config-replication-design.md §6.5/§7.1). _(From FUTURE_WORK.md §1 + §8, 2026-05-12.)_
|
||||
- [ ] primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
|
||||
- [ ] read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
|
||||
- [ ] entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs is already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion.
|
||||
|
||||
## Hardening / Direct-Implementation Items
|
||||
|
||||
Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.
|
||||
|
||||
- [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_
|
||||
- [ ] validate_host() real authorization — currently a stub returning success for all peers; MUST be implemented before `CONF_ENABLE_UPGRADE_CONFIG=1`. Minimum: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_
|
||||
- [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_
|
||||
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
|
||||
- [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_
|
||||
- [ ] steering sync — `ha-async-io.md` still documents CTRL_PRIMARY_COMMITTING and the two-tier model; update after ha-consistency-invariants lands.
|
||||
|
||||
## Superseded / Absorbed Items
|
||||
|
||||
- ~~Quorum-ACK gate before config restart~~ _(FUTURE_WORK.md §2)_ — **largely superseded by ha-consistency-invariants Requirement 5**: configuration changes now require majority acknowledgment before being confirmed/committed, and fail to the requester on timeout (no silent rollback when standbys are offline). Residual to verify in that spec's design: restart sequencing after a committed config change (don't restart the primary before the commit barrier passes).
|
||||
- ~~Lost-and-found keyword validation~~ _(FUTURE_WORK.md §8)_ — folded into the admin-config-tool backlog item above.
|
||||
|
||||
## Notes
|
||||
|
||||
- Backlog items here are seeds for `/kiro-discovery` / `/kiro-spec-init`; promote an item by creating its spec under `.kiro/specs/`.
|
||||
- Provenance: items marked _(FUTURE_WORK.md §N)_ originate from the 2026-05-12 config-replication-design.md review, merged here 2026-06-10.
|
||||
|
|
@ -10,7 +10,7 @@ xcecli (Go/Cobra) ──REST──► apiserver (Go/Gin) ──CGO──► C da
|
|||
|
||||
Request path inside apiserver: `Gin Router → Middleware (CORS/RateLimit/IPWhitelist/JWT) → RBAC Check → Controller → Services → In-Memory Store`
|
||||
|
||||
All controller state is in-memory, seeded at startup from YAML config. No persistent database in the API tier. C daemons communicate via libtirpc RPC; WAL + checkpoint provide persistence at the controller level.
|
||||
All controller state is in-memory, seeded at startup from YAML config. No persistent database in the API tier. C daemons communicate via TCP using a custom messaging layer: iomgr (epoll-based I/O loop), vtaskmgr (thread pool for packet dispatch), and xce_request/xce_reply for non-blocking send/receive. WAL + checkpoint provide persistence at the controller level.
|
||||
|
||||
## Core Technologies
|
||||
|
||||
|
|
@ -21,7 +21,7 @@ All controller state is in-memory, seeded at startup from YAML config. No persis
|
|||
- **Zap v1.27.0** — structured logging throughout (both Go modules)
|
||||
- **Viper v1.19.0 / v1.20.1** — YAML config management
|
||||
- **Cap'n Proto** — wire serialization for WAL and daemon-to-daemon replication
|
||||
- **libtirpc** — inter-daemon RPC transport
|
||||
- **iomgr / vtaskmgr / delaymgr** — libxce async I/O framework: single-threaded epoll loop (iomgr), thread-pool packet dispatch (vtaskmgr), timer thread for periodic/blocking tasks (delaymgr)
|
||||
|
||||
## Build Systems
|
||||
|
||||
|
|
@ -97,4 +97,4 @@ Early return on every error. Log with zap fields before returning.
|
|||
3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell.
|
||||
4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin.
|
||||
5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output.
|
||||
6. **Single-writer HA** — controller is always one active primary; standby election via LSF-style priority, not consensus.
|
||||
6. **Single-writer HA** — controller is always one active primary; standby election via Raft-style consensus (term-based voting, freshness gates) in `raft.c`, with the single-writer constraint avoiding Raft log replication complexity.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue