update documents
This commit is contained in:
parent
c705cadadc
commit
d70bf6a402
51 changed files with 995 additions and 1997 deletions
|
|
@ -48,7 +48,7 @@ Every HTTP response uses a consistent envelope via two helpers in `controllers/r
|
|||
```go
|
||||
// Success (always HTTP 200)
|
||||
ReturnSuccess(ctx, code, msg, data)
|
||||
// → {code: 0, msg: "success", data: {...}}
|
||||
// → {code: 200, msg: "success", data: {...}}
|
||||
|
||||
// Error (HTTP status == code parameter)
|
||||
ReturnError(ctx, code, msg, details)
|
||||
|
|
@ -61,7 +61,7 @@ All codes defined in `common/common.go`:
|
|||
|
||||
| Code | Constant | Usage |
|
||||
|------|----------|-------|
|
||||
| 0 | `SUCCESS_CODE` | All success responses |
|
||||
| 200 | `SUCCESS_CODE` | All success responses |
|
||||
| 400 | `INVALID_PARAMS_CODE` | Malformed/missing input |
|
||||
| 401 | `ERROR_AUTH_CHECK_TOKEN_CODE` | Missing/invalid JWT |
|
||||
| 403 | `ERROR_AUTH_CODE` | Valid JWT, insufficient permission |
|
||||
|
|
@ -79,7 +79,7 @@ WhitelistURL → WhitelistIP → RateLimit → ValidateHandler (JWT) → CORS
|
|||
|
||||
- **WhitelistURL**: filter by allowed paths (applied first for efficiency)
|
||||
- **WhitelistIP**: abort 403 if client IP not in whitelist
|
||||
- **RateLimit**: 10 req/sec, burst 20 (global config)
|
||||
- **RateLimit**: 1000 req/sec, burst 2000 (global config, `config/app.yaml`)
|
||||
- **ValidateHandler**: parse and validate JWT from `Authorization: Bearer` header; validate body content-type
|
||||
- **CORS**: origin checks via config
|
||||
- **RBAC**: `RequirePermission("action:scope")` per-route, enforced by `controllers/rbac_check.go`
|
||||
|
|
@ -96,13 +96,13 @@ Defined in `config/rbac.yaml`. Roles support inheritance via `includes:`.
|
|||
All inter-daemon messages are Cap'n Proto serialized. Schema: `src/schema/controller.capnp`.
|
||||
|
||||
**Request** (`xce_request_t`):
|
||||
- `opcode` — OPCode enum (38 values)
|
||||
- `opcode` — OPCode enum (43 values)
|
||||
- `seq` — request sequence number
|
||||
- `from` — sender peer info (host/key)
|
||||
- `body` — union of 27 message types
|
||||
- `body` — union of 29 message types (including the empty `nomsg` placeholder)
|
||||
|
||||
**Reply** (`xce_reply_t`):
|
||||
- `opcode` — always `OPCode_reply` or `OPCode_redirect`
|
||||
- `opcode` — echoes the originating request's opcode by default (`init_reply_by_req()` sets `reply->opcode = req->opcode`); many call sites override it with a type-specific opcode (e.g. `OPCode_preVoteReply`, `OPCode_walAck`, `OPCode_schedAdmin`). Consumers dispatch on this value, so it is not a fixed `OPCode_reply`/`OPCode_redirect` pair — that pairing only applies to the separate wrapper `xce_request_t` that `xce_reply()` builds internally.
|
||||
- `status` — error code (0 = success)
|
||||
- `seq` — echoes request seq
|
||||
- `data` — union of reply payloads (clusterconf, jobid, stateReport, walackreplymsg, etc.)
|
||||
|
|
@ -111,7 +111,7 @@ All inter-daemon messages are Cap'n Proto serialized. Schema: `src/schema/contro
|
|||
|
||||
| Group | OPCodes | Purpose |
|
||||
|-------|---------|---------|
|
||||
| Control | `jobsubmit (11)`, `jobdispatch (8)`, `jobcontrol (15)` | Job lifecycle |
|
||||
| Control | `jobsub (11)`, `jobdispatch (8)`, `jobcontrol (15)` | Job lifecycle |
|
||||
| Cluster | `hello (2)`, `join (3)`, `clusterconf (17)`, `heartbeat (25)` | Membership and liveness |
|
||||
| Replication | `replConfig (29)`, `replConfigPush (30)`, `walCatchupEntry (37)`, `snapshotTransfer (36)` | Config + WAL replication |
|
||||
| Vote | `requestVote`, reply | Raft election |
|
||||
|
|
@ -137,8 +137,8 @@ int xce_decode(io_packet_t *pkt, xce_request_t *req);
|
|||
|
||||
### Adding a New Opcode
|
||||
|
||||
1. Add the opcode constant to `src/include/request.h` (OPCode enum)
|
||||
1. Add the opcode constant to the `OPCode` enum in `src/schema/controller.capnp`
|
||||
2. Add the message struct to `src/schema/controller.capnp` (Request/Reply union arms)
|
||||
3. Rebuild capnp schemas (automatic via meson)
|
||||
4. Wire the handler in the dispatch gate (`ha_dispatch_gate` in `ha.c`)
|
||||
4. Add a `case OPCode_...:` in the `switch (request.opcode)` dispatch inside `myreq_handler()` in `cluster.c` (this runs after `ha_dispatch_gate`, which only applies a role-based allow/deny filter, not per-opcode handler logic)
|
||||
5. Add encode/decode paths in `src/lib/request.c` if using raw payloads
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
## Philosophy
|
||||
|
||||
A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Every build includes debug symbols + ASAN. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management.
|
||||
A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Only the `asan` profile (the default for `/xce-build`) includes both debug symbols and ASAN; `debug` builds symbols without ASAN, and `release` builds (including the nix `xcecore` production package) carry neither. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management.
|
||||
|
||||
## Build Commands
|
||||
|
||||
|
|
@ -16,6 +16,7 @@ All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
|
|||
/xce-build worker # lib + worker (daemon only, no unit tests)
|
||||
/xce-build apiserver # lib + apiserver + xcecli + Go unit tests
|
||||
/xce-build xcecli # xcecli + Go tests
|
||||
/xce-build xceadmin # xceadmin (meson-driven; builds src/xceadmin's venv/tests)
|
||||
```
|
||||
|
||||
`controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it.
|
||||
|
|
@ -23,7 +24,7 @@ All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
|
|||
Each invocation:
|
||||
1. `nix develop` into the hermetic shell
|
||||
2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build)
|
||||
3. `ninja -C <dir>` (compile all + test executables)
|
||||
3. `meson compile -C <dir> [targets...]` — per-component invocations pass an explicit target list scoped to just that component (e.g. `controller` compiles only `xce_controller controller_unit_test controller/xce_test`); only `/xce-build all` compiles with no target args, building everything
|
||||
4. Run registered meson tests (skipped for non-`asan` profiles — see below)
|
||||
|
||||
**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.
|
||||
|
|
@ -41,9 +42,10 @@ src/
|
|||
├── controller/meson.build # xce_controller daemon + controller_unit_test
|
||||
├── worker/meson.build # xcecd daemon
|
||||
├── scheduler/meson.build # scheduler daemon + test
|
||||
├── autoscaler/meson.build # autoscaler daemon
|
||||
├── autoscaler/meson.build # libscaler static lib + provider tools (no daemon binary)
|
||||
├── frontend/apiserver/meson.build # Go apiserver via tools/go_build
|
||||
└── frontend/xcecli/meson.build # Go xcecli via tools/go_build
|
||||
├── frontend/xcecli/meson.build # Go xcecli via tools/go_build
|
||||
└── xceadmin/meson.build # xceadmin config-build tool (used by /xce-deploy-ha)
|
||||
```
|
||||
|
||||
**Dependency flow**: `lib → {controller, worker, scheduler, apiserver}`
|
||||
|
|
@ -51,8 +53,7 @@ src/
|
|||
**Test registration**:
|
||||
```python
|
||||
# controller/meson.build
|
||||
test('controller',
|
||||
controller_unit_test,
|
||||
test('controller test', controller_unit_test,
|
||||
suite: 'controller',
|
||||
timeout: 300,
|
||||
env: ['ASAN_OPTIONS=detect_leaks=0'])
|
||||
|
|
@ -71,7 +72,7 @@ nix build .#xcecore_debug # default package: debug + ASAN
|
|||
nix build .#xcecli # CLI only
|
||||
```
|
||||
|
||||
**AppImage**: The flake produces an AppImage wrapping `xce` (shell entry point) with bundled daemons, haproxy, glow, and less. The entry script reads from `/tools/appimage/entry.sh`.
|
||||
**AppImage**: `nix build .#xcecore`/`.#xcecore_debug` do not produce an AppImage — they only install `xce` (a copy of `tools/appimage/entry.sh`, see `src/meson.build`) as a plain shell entry-point binary. The `xce` flake package (`appImageEntryPackage` in `src/flake.nix`) wraps that entry script with bundled daemons, haproxy, glow, and less; the actual `.AppImage` file is produced separately with `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` (see README.md, "3.5 build AppImage package").
|
||||
|
||||
## Local HA Cluster Deployment
|
||||
|
||||
|
|
@ -86,7 +87,7 @@ This creates under `/tmp/xce-deploy/`:
|
|||
certs/ # TLS certs for all peers
|
||||
controller/
|
||||
ctrl-1/
|
||||
conf/{current, cluster.bin} # primary's config
|
||||
conf/{current, cluster.bin.<tag>} # primary's config (tag = sha256sum prefix)
|
||||
data/ # WAL + checkpoint storage
|
||||
ctrl-2/
|
||||
conf/ # receives config via OPCode_replConfigPush
|
||||
|
|
@ -135,7 +136,7 @@ Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` f
|
|||
| Controller daemon | `nix build .#xcecore` | `xce_controller` binary |
|
||||
| Worker daemon | same | `xcecd` binary |
|
||||
| Scheduler daemon | same | `xce_scheduler` binary |
|
||||
| REST API server | same | `apiserver` binary |
|
||||
| REST API server | same | `xce_apiserver` binary |
|
||||
| CLI tool | `nix build .#xcecli` | `xcecli` binary |
|
||||
| AppImage | `nix build .#xcecore` | `xce-x86_64.AppImage` |
|
||||
| AppImage | `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` | `.AppImage` bundle (see README.md 3.5) |
|
||||
| Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries |
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ Fail fast with a consistent, machine-readable response. Handle errors at the con
|
|||
|
||||
## Go API Response Pattern
|
||||
|
||||
All HTTP responses go through two helpers in `controllers/response.go`:
|
||||
All HTTP responses go through two helpers in `controllers/response.go` (a small number of legacy handlers, e.g. `FilesController` list/dir endpoints in `controllers/files.go`, still call `ctx.JSON()` directly — treat those as bugs to fix opportunistically, not as sanctioned exceptions):
|
||||
|
||||
```go
|
||||
// Success (always HTTP 200)
|
||||
|
|
@ -18,7 +18,7 @@ ctx.Abort() // always Abort() after ReturnError in middleware
|
|||
return // always return immediately after ReturnError in handlers
|
||||
```
|
||||
|
||||
**The early-return pattern is non-negotiable.** Every error path ends with `ReturnError(...); return`.
|
||||
**The early-return pattern is non-negotiable.** Every error path ends with `ReturnError(...); return`. `ctx.Abort()` should always follow in middleware since gin does not stop the handler chain on its own — `middleware/whitelist.go`'s `WhitelistIPMiddleware` currently has one path that returns without calling `ctx.Abort()` (invalid client IP); that is a gap to close, not an accepted variant.
|
||||
|
||||
## HTTP Status Code Mapping
|
||||
|
||||
|
|
@ -78,17 +78,23 @@ if (rc < 0) {
|
|||
|
||||
## Input Validation Pattern
|
||||
|
||||
Validate at the controller boundary using a `preCheck{Resource}Request()` helper before touching services:
|
||||
Validate at the controller boundary in two steps: bind the request body with `utils.BindJSONStrict`, then pass the already-bound struct pointer to a `preCheck{Resource}Request()` helper before touching services:
|
||||
|
||||
```go
|
||||
func (c JobController) SubmitJob(ctx *gin.Context) {
|
||||
req, err := preCheckSubmitJobRequest(ctx)
|
||||
func (j JobController) CreateJob(ctx *gin.Context) {
|
||||
var req models.JobSubmitRequest
|
||||
if err := utils.BindJSONStrict(ctx, &req); err != nil {
|
||||
ReturnError(ctx, common.ERROR_CODE, err.Error(), nil)
|
||||
return
|
||||
}
|
||||
|
||||
parsedIndices, err := preCheckJobRequest(&req)
|
||||
if err != nil {
|
||||
ReturnError(ctx, common.INVALID_PARAMS_CODE, err.Error(), nil)
|
||||
ReturnError(ctx, common.ERROR_CODE, err.Error(), nil)
|
||||
return
|
||||
}
|
||||
// ... service call
|
||||
}
|
||||
```
|
||||
|
||||
Keep `preCheck*` functions in the same file as the handler. They bind, validate, and return the typed request struct or an error.
|
||||
Keep `preCheck*` functions in the same file as the handler. They take the already-bound request struct pointer (never `ctx`) and return `error`, or `(extra, error)` when the caller also needs derived data — e.g. `preCheckJobRequest(req *models.JobSubmitRequest) ([]int, error)`, `preCheckHostControlRequest(req *models.HostControlRequest) (bool, error)`. They never bind the request themselves and never return the request struct — binding is always the separate `utils.BindJSONStrict` step shown above. Note that handlers choose the response code for bind/precheck failures per-endpoint; `common.ERROR_CODE` (500) and `common.INVALID_PARAMS_CODE` (400) both appear in practice depending on the handler.
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ Three components in `src/lib/` plus a dedicated slow-lane executor:
|
|||
| Component | Thread model | May do | Must NOT do |
|
||||
|-----------|-------------|--------|-------------|
|
||||
| **iomgr** | Single-threaded epoll loop | non-blocking socket I/O; endpoint lifecycle; **dispatch to vtask** | any blocking call; **any inline message processing** (dispatch it) |
|
||||
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1082) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool |
|
||||
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1743-1745) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool |
|
||||
| **delaymgr** | Single timer thread | fire timers; **trigger** (enqueue) work; return fast | **connect(), file I/O, any blocking** — it drives heartbeats + election timeouts |
|
||||
| **slow lane** | dedicated executor / admission-capped pool | long blocking I/O: reconnect, snapshot transfer, config-push | — (this is where blocking ops go, NOT delaymgr) |
|
||||
|
||||
|
|
@ -23,10 +23,10 @@ Three components in `src/lib/` plus a dedicated slow-lane executor:
|
|||
No thread that carries liveness signals (iomgr, delaymgr) may execute blocking work inline:
|
||||
1. **iomgr** does only non-blocking I/O + dispatch. It never blocks and never processes a message inline — it hands off to a vtask.
|
||||
2. **delaymgr** only fires timers and triggers work, then returns. It NEVER blocks (no `connect()`, no file I/O) — because the same single thread drives heartbeats and election timeouts, so any block there stalls liveness for the whole cluster and can cause spurious failover.
|
||||
3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, commit.c:195-203) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here.
|
||||
3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, `wait_cap`/`wait_admit_locked`, commit.c:241-271) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here.
|
||||
4. **Long blocking I/O** (reconnect, snapshot transfer, config-push) runs on a dedicated slow lane / admission-capped executor, never on iomgr or delaymgr.
|
||||
|
||||
> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`) — tracked as the `raft-drive-async` / P2 work in roadmap.md.
|
||||
> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`). **All three are now fixed** — spec `raft-drive-async` (IMPLEMENTED, validate-impl GO 2026-07-18, 21/21 tasks) moved each off its liveness thread: snapshot transfer now runs on a dedicated slow-lane worker (`snapshot_lane_fn`, `ha.c:4021`, kept off the iomgr thread — see the "DEFAULT ON" note at `ha.c:5568`); `push_config_transport_only` now spawns a dedicated pthread per standby (`pthread_create`, `config_repl.c:1589`) instead of busy-polling+joining on iomgr; and `socket_connect` (`netio.c:302`, doc comment `netio.c:287`) is now a bounded, non-blocking TCP connect via `poll()` with `connect_timeout_ms`, so it no longer pins the delaymgr heartbeat thread. See bug-1447 below for a same-day crash this spec's own validation soak surfaced and fixed.
|
||||
|
||||
Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.
|
||||
|
||||
|
|
@ -87,9 +87,9 @@ The **durable-majority barrier + the own-term `WAL_NOOP` writable gate** (`commi
|
|||
|
||||
| Role | Allowed |
|
||||
|------|---------|
|
||||
| **CTRL_PRIMARY** | All opcodes (client ops subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` is still admitted) |
|
||||
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, + all replication opcodes |
|
||||
| **CTRL_INIT / ELECTING** | requestVote, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer |
|
||||
| **CTRL_PRIMARY** | All opcodes, subject to two fences checked before the quorum gate (`ha_dispatch_gate`, ha.c:4525): `config_serve_fenced()` (ha-term-safety task 4.3) and `upgrade_materialization_fenced()` (config-upgrade-snapshot task 6.2) each refuse state-changing ops while set, admitting only replication/read-only/status opcodes. Once unfenced: client ops are subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` (plus read-only/status/replication opcodes) is still admitted |
|
||||
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, heartbeat, redirect, steppingDown, requestVote, preVoteRequest, probeConfigTag, upgradeConfigAck, ctrlstatus, + all replication opcodes (replEntry, replAck, replSnapshot, replCatchUpDone, replConfig, replConfigPush, walAck, snapshotTransfer, walCatchupEntry) |
|
||||
| **CTRL_INIT / ELECTING** | requestVote, preVoteRequest, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer, ctrlstatus |
|
||||
|
||||
This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.
|
||||
|
||||
|
|
|
|||
|
|
@ -6,9 +6,9 @@ XCE is an HPC (High-Performance Computing) cluster management system providing j
|
|||
|
||||
1. **Job lifecycle management** — submit, start, stop, restart, kill, and monitor jobs including job arrays. Resource usage data is collected and exposed via API.
|
||||
2. **Queue and scheduling** — plugin-based scheduling engine for job-to-node placement with queue control (enable/disable) and expression-based resource filtering.
|
||||
3. **Cluster and host management** — node status monitoring (online/offline/unavailable), cluster topology tracking, and host control operations.
|
||||
4. **Multi-tenant RBAC + pluggable auth** — domain-based `action:scope` permission model (e.g., `job:view:self`). Auth backends: configfile, LDAP, OAuth2, PAM. Built-in IP whitelist and rate limiting.
|
||||
5. **Controller persistence and HA** — WAL + checkpoint-based single-controller persistence with LSF-style priority election for standby failover and live replication over TCP (Cap'n Proto serialization).
|
||||
3. **Cluster and host management** — node status monitoring (OK, UNAVAIL, LOCKEDU, LOCKEDW, BUSY, CLOSED, CLOSED_BY_ADMIN), cluster topology tracking, and host control operations.
|
||||
4. **Multi-tenant RBAC + pluggable auth** — two-part `action:operation` permission IDs (e.g., `job:view`) with a separately-keyed domain scope per role (e.g., `type: self`, `type: self_ug`, `type: all`). Auth backends: configfile, LDAP, OAuth2, PAM. Built-in IP whitelist and rate limiting.
|
||||
5. **Controller persistence and HA** — WAL + checkpoint-based single-controller persistence with Raft-style quorum election (RequestVote, term-based voting, freshness gates) for standby failover and live replication over TCP (Cap'n Proto serialization).
|
||||
|
||||
## Target Use Cases
|
||||
|
||||
|
|
@ -55,9 +55,11 @@ raft-sim-conformance — all now GO. That sweep surfaced one real bug
|
|||
(bug-1447: same-batch endpoint use-after-free in `io_worker`'s epoll
|
||||
dispatch loop, see `ha-async-io.md`), root-caused and fixed the same day.
|
||||
|
||||
Remaining backlog (requirements-only or design-pending, not yet implemented):
|
||||
admin-config-tool (spec finalized), scheduler-framework, config-upgrade-snapshot
|
||||
(blocked on entity-bearing snapshots). See `roadmap.md` for dependency order
|
||||
Remaining backlog (requirements-only, design-pending, or partially implemented):
|
||||
admin-config-tool (spec finalized), config-upgrade-snapshot (tasks generated,
|
||||
unblocked, implementation nearly complete). scheduler-framework was decomposed
|
||||
into scheduler-core (implementation-complete), scheduler-allocation, and
|
||||
scheduler-preemption (both active). See `roadmap.md` for dependency order
|
||||
and status.
|
||||
|
||||
Key design decisions already locked:
|
||||
|
|
|
|||
|
|
@ -6,10 +6,10 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin
|
|||
|
||||
## In Flight
|
||||
|
||||
- **scheduler-framework** — NEW 2026-07-06 (`/kiro-discovery`). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: `.kiro/specs/scheduler-framework/brief.md`. Dependencies: none hard (builds on existing plugin framework + select expression evaluator). Next: `/kiro-spec-requirements scheduler-framework`.
|
||||
- ~~**scheduler-framework** — NEW 2026-07-06 (`/kiro-discovery`). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: `.kiro/specs/scheduler-framework/brief.md`.~~ **DECOMPOSED 2026-07-27** into three child specs (`.kiro/specs/scheduler-framework/spec.json` `child_specs`): **scheduler-core** — core/driver separation, pluggable job picker interface, and simulation mode; **IMPLEMENTATION-COMPLETE**, GO/NO-GO validation 2026-08-07/08 (2 non-blocking follow-ups), 45/45 sub-tasks done. **scheduler-allocation** — lazy iterator-driven allocation architecture, parallel job support, session-level unsatisfiable resreq cache; requirements generated, `blocked_on: [scheduler-core]` — now unblocked. **scheduler-preemption** — SIGSTP-based victim selection + controller→worker pipeline; requirements generated, `blocked_on: [scheduler-core]` — now unblocked. The parent `scheduler-framework` spec is retained only for discovery-context provenance and is no longer in flight itself. Next: `/kiro-spec-design scheduler-allocation` and `/kiro-spec-design scheduler-preemption`.
|
||||
- ~~**ha-consistency-invariants** — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.~~ **COMPLETE 2026-07-13** — all four sub-specs (ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) implemented; Criterion 2.9 (Facts-Apply invariant) satisfied via raft-apply-decoupling. No longer in flight.
|
||||
- ~~ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs `/xce-deploy-ha` cluster); sub-spec 4's E2E extends it~~ **task 5.4 CLOSED 2026-07-18** via `/kiro-validate-impl`'s live 10-round sequential soak (972/972 tests, zero loss). Spec is GO, no longer in flight.
|
||||
- ~~**libxce-failover** — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) `xce_request_with_retry()` primitive + adds multi-master rotation; classifies on the real controller status taxonomy `E_XCE_REDIRECT` / `E_XCE_NOMASTER` / `E_XCE_NOT_READY` / `E_XCE_STANDBY` (`errnox.h:22-25`); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, `controller.capnp:568-569`, just unpopulated in `join.c`). Verified E2E via `xce_test` (the original reproduction). Closes the integration gap left by `ha-config-bootstrap` Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).~~ **IMPLEMENTED, validate-impl GO 2026-07-18** (all 19 tasks were done since 2026-06-18; the `ready_for_implementation` flag was just never flipped — administrative staleness, not a real block). No longer in flight. `component-failover-adoption` (backlog, below) is the next layer on top of this.
|
||||
- ~~**libxce-failover** — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) `xce_request_with_retry()` primitive + adds multi-master rotation; classifies on the real controller status taxonomy `E_XCE_REDIRECT` / `E_XCE_NOMASTER` / `E_XCE_NOT_READY` / `E_XCE_STANDBY` (`errnox.h:22-25`); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, `controller.capnp:598-599`, just unpopulated in `join.c`). Verified E2E via `xce_test` (the original reproduction). Closes the integration gap left by `ha-config-bootstrap` Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).~~ **IMPLEMENTED, validate-impl GO 2026-07-18** (all 19 tasks were done since 2026-06-18; the `ready_for_implementation` flag was just never flipped — administrative staleness, not a real block). No longer in flight. `component-failover-adoption` (backlog, below) is the next layer on top of this.
|
||||
|
||||
## Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21)
|
||||
|
||||
|
|
@ -21,9 +21,11 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin
|
|||
|
||||
> **ha-term-safety** (`.kiro/specs/ha-term-safety/`) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the **config-change marker** (R7 — config changes as `(term,lsn)` log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live **bug-851 chaotic-failover regression**. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). **Scope: R1–R7 only.** The former R8 (config-upgrade full-snapshot consistency, bug-857) was **descoped 2026-06-21 to the independent `config-upgrade-snapshot` spec** (below), because it requires entity-bearing snapshots that do not exist yet. **IMPLEMENTED, validate-impl GO 2026-07-17** (20/20 tasks; closed bug-1445, a first-boot promotion-check race with job-array-single-wal-record's Live Apply Resource Fence, as an out-of-boundary follow-up fix).
|
||||
|
||||
## Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`)
|
||||
> **raft-adapter-conformance** (`.kiro/specs/raft-adapter-conformance/`) — NEW 2026-07-29, sibling audit to the library-side conformance work above: a systematic, action-by-action pass over the controller's Raft **adapter** (the wire-level send/receive/reply code in `ha.c`/`repl.c`/`config_repl.c`) against `raft.tla`'s actions — distinct from `raft-sim-conformance`'s in-process library harness. Motivated by a live-soak finding that the controller's `WalAckReply` constructor hardcoded success (no failing reply ever reached the primary) plus a standby catch-up streaming-range self-deadlock, both already fixed (bug-1479) before this spec was seeded. This pass found and fixed two more instances of the same defect class: configuration-push rejects previously sent no reply at all (bug-1480), and a vote/PreVote endpoint failure arriving *after* connect routed through a legacy tally path instead of the production one (extending bug-1423's connect-time fix to that remaining timing window). Also records, as a single explicit decision, which `AppendEntries` mechanism is production-authoritative (`repl_broadcast` + `ha_stream_wal_to_standby`; the per-peer index-driven `raft_send_append_entries` stays dormant by documented design, not oversight). Requirements/design/tasks generated and approved, all 15 leaf tasks (1–7.3) checked; `src/controller/CLAUDE.md`'s Protocol Verification Checklist marks the corresponding invariants "post `raft-adapter-conformance`, landed 2026-07-29" — `spec.json` phase (`tasks-generated`) has not been flipped to reflect this, same administrative-staleness pattern seen elsewhere in this doc (cf. the former libxce-failover entry above). Dependencies: raft-log-replication, raft-commit-safety, ha-term-safety, raft-sim-conformance (adjacent/informational — this spec fixes the adapter layer those validate underneath).
|
||||
|
||||
**Goal:** make the controller follow canonical Raft ordering — *pre-hooks → raft commit → apply → notify* — with the raft protocol step lifted OUT of the notification hook fan-out. Today `persist_job_hook_fun` (an `AFTER_EVENT` hook) performs the WAL append + synchronous majority-commit barrier inline, and `create_job()` mutates in-memory state BEFORE that commit (`job.c:558` before `job.c:593`; `persist.c:2532`/`persist.c:2198`/`ha.c:3080`). A failed/timed-out commit does not cleanly revert the optimistic mutation — the structural root of the bug-795/828/851/1060 divergence family. Decision (user, 2026-06-24): **apply-after-commit (canonical)** — a single shared `apply_committed_entry()` seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decompose via Path E (2 new specs + touchpoints).
|
||||
## Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`); IMPLEMENTED — see `raft-apply-decoupling` below
|
||||
|
||||
**Goal (historical — achieved):** make the controller follow canonical Raft ordering — *pre-hooks → raft commit → apply → notify* — with the raft protocol step lifted OUT of the notification hook fan-out. At the time this family was opened, `persist_job_hook_fun` (an `AFTER_EVENT` hook) performed the WAL append + synchronous majority-commit barrier inline, and `create_job()` mutated in-memory state BEFORE that commit — the structural root of the bug-795/828/851/1060 divergence family. A failed/timed-out commit did not cleanly revert the optimistic mutation. Decision (user, 2026-06-24): **apply-after-commit (canonical)** — a single shared `apply_committed_entry()` seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decomposed via Path E (2 new specs + touchpoints), delivered as `raft-apply-decoupling` (**IMPLEMENTED 2026-07-07**, see below). Current code reflects the fix: `persist_job_hook_fun` (`persist.c` ~6509-6553) now explicitly returns early for `WAL_MODE_WAIT` entries, with a comment noting job-submit durability is invoked directly by `handle_jobsubmit` — `persist_submit_job_payloads_and_commit()`, called strictly before any job object is constructed — never from this hook; `create_job()` is now defined at `job.c:48`.
|
||||
|
||||
### Async-drive prerequisites (surfaced 2026-06-25 by blocking audit; thread-role contract now in `ha-async-io.md`)
|
||||
|
||||
|
|
@ -60,34 +62,34 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
|
|||
|
||||
## Future Specs / Backlog (dependency order)
|
||||
|
||||
- [ ] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **HARD dependency: `entity-bearing-snapshots`** (today snapshots are a 16-byte LSN sentinel; the entity payload does not exist) AND `ha-term-safety` (the config-change marker it anchors on). Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). Promote to design once `entity-bearing-snapshots` lands.
|
||||
- [ ] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **Former HARD dependency `entity-bearing-snapshots` is CLEARED** — it shipped as spec `ha-entity-snapshots` (**IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18**; see Future Specs/Backlog below), and `ha-term-safety` (the config-change marker it anchors on) is also implemented. `config-upgrade-snapshot`'s own `spec.json` now has `blocked_on: []`. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). **Ready to promote to design.**
|
||||
- [ ] component-failover-adoption — **recorded 2026-06-16; depends on libxce-failover (HARD).** Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link `libxce_dep`; apiserver uses it via cgo, `src/frontend/apiserver/backend/clib.go`), but two per-component integration items remain: (1) each component must hand libxce the **full master list** instead of a single endpoint (today even the CLI uses only `conf->masters[0]`, `client.c:122`); (2) bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons (worker `controller_connector.c`, scheduler `handler.c`) this must be verified **liveness-safe**, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. (`xce_test` is NOT here — it is the in-spec verification client of libxce-failover.) Promote via `/kiro-spec-init component-failover-adoption` after libxce-failover lands.
|
||||
- [ ] worker-rejoin-resync — **planned long ago, no effort yet (recorded 2026-06-10).** When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): `xce_join_req` carries only `hostattrs_t` (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the `mark_job_*` idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. **Elevated 2026-06-11**: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only).
|
||||
- [ ] **admin-config-tool** — **PROMOTED from backlog to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/admin-config-tool/brief.md`). **NEW: `xceadmin` Python CLI** for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: **(a) `config build`** — directory scan: parse `configs/{hosts,queues,resources,users}.yaml`, validate against JSON schemas (`configs/schema/*.schema.json`), serialize `ClusterConfig` capnp struct to `cluster.bin` with SHA-256; **(b) `config push`** — zero-config dev (default `localhost:9991`, same as xce_test), single `--controller` flag for production, auto-discovers peers + primary from controller reply, sends `OP_upgradeConfig`, displays `{versionTag,ackedCount,error}`. **Single dependency: `controller.capnp`.** Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (`pip install`). The tool is the client-side trigger for `config-upgrade-snapshot`'s controller-side path. MUST validate against reserved name `"lost_and_found"` for queue/host (defence-in-depth). xcecli comparison: `xcecli` = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); `xceadmin` = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: `xcecommon` Python package with capnp schema + wire protocol reusable by both tools. Dependencies: `controller.capnp` schema only. **Blocks: nothing — ships independently.** Controller-side upgrade snapshot path (`config-upgrade-snapshot`) remains blocked on `entity-bearing-snapshots` (pre-existing). Next: `/kiro-spec-requirements admin-config-tool`.
|
||||
- [ ] **admin-config-tool** — **PROMOTED from backlog to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/admin-config-tool/brief.md`). **NEW: `xceadmin` Python CLI** for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: **(a) `config build`** — directory scan: parse `configs/{hosts,queues,resources,users}.yaml`, validate against JSON schemas (`configs/schema/*.schema.json`), serialize `ClusterConfig` capnp struct to `cluster.bin` with SHA-256; **(b) `config push`** — zero-config dev (default `localhost:9991`, same as xce_test), single `--controller` flag for production, auto-discovers peers + primary from controller reply, sends `OP_upgradeConfig`, displays `{versionTag,ackedCount,error}`. **Single dependency: `controller.capnp`.** Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (`pip install`). The tool is the client-side trigger for `config-upgrade-snapshot`'s controller-side path. MUST validate against reserved name `"lost_and_found"` for queue/host (defence-in-depth). xcecli comparison: `xcecli` = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); `xceadmin` = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: `xcecommon` Python package with capnp schema + wire protocol reusable by both tools. Dependencies: `controller.capnp` schema only. **Blocks: nothing — ships independently.** Controller-side upgrade snapshot path (`config-upgrade-snapshot`) was blocked on `entity-bearing-snapshots`; that blocker is now cleared (see Future Specs/Backlog above — `ha-entity-snapshots` implemented, `config-upgrade-snapshot` `blocked_on: []`). Next: `/kiro-spec-requirements admin-config-tool`.
|
||||
- [ ] lightweight rejoin re-attach (failover optimization) — **recorded 2026-06-16; deferred (YAGNI).** Today, on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale. Dependencies: libxce-failover (rejoin hook), component-failover-adoption (daemon rejoin content), worker-rejoin-resync (worker job state).
|
||||
- [x] job-array-single-wal-record — **IMPLEMENTED, validate-impl GO 2026-07-17** (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). **PROMOTED to a spec 2026-06-24** (`/kiro-discovery`; brief at `.kiro/specs/job-array-single-wal-record/brief.md`; now tracked under the *Raft Apply-Ordering & Hook Decoupling* phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready.
|
||||
- [ ] bulk job-array query/cancel — **recorded 2026-07-13, deferred out of `job-array-single-wal-record` scope by design.** A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for `job-array-single-wal-record` — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array *submission* and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: `job-array-single-wal-record` (per-element identity/addressing this would operate over). Promote via `/kiro-discovery` once job-array-single-wal-record ships.
|
||||
- [x] ~~WAL group-commit batching~~ — **recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal).** The `/kiro-validate-design` review surfaced that the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible and not covered by `job-array-single-wal-record` — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): **do group-commit now** — repurpose `fsync_worker` from a timer into a demand-driven flusher (batch one `fdatasync` across pending appends, advance `durable_lsn`, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
|
||||
- [ ] primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
|
||||
- [ ] read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
|
||||
- [ ] entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs is already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. **Downstream (recorded 2026-06-21): `config-upgrade-snapshot` HARD-depends on this** — it needs an entity-bearing snapshot to deliver an identical materialized post-upgrade state to every member (bug-857). **This is also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0) — recorded 2026-06-25:** WAL streaming alone cannot serve a standby whose needed range was trimmed (bug-795/828/1147), and a new node always starts empty, so safe aggressive WAL trimming + new-node join *requires* InstallSnapshot (snapshot-at-boundary + tail WAL). Until this lands, trim must stay cursor-bounded (strategy-A stopgap, see Hardening) and the system cannot safely admit a standby that has fallen below the trim floor. Scope to add when promoted: the standby-side snapshot-install / apply path (request snapshot when its cursor is below the leader's earliest retained LSN, install, resume from snapshot tail).
|
||||
- [x] entity-bearing snapshots ("Task 11" format) — **PROMOTED to spec `ha-entity-snapshots` (created 2026-07-01); IMPLEMENTED, validate-impl GO 2026-07-18** (21/21 tasks 1.1-8.1; 962 controller tests green including all 9 `test_persist_snapshot_entities.c` cases; live `--force-snapshot` soak: committed_divergence=0, committed_job_loss=0, confirmed real "HA: snapshot" install log line). `write_snapshot()` (`persist.c` ~2304-2372) now collects and serializes job/queue/host entity frames (`collect_job_frames`/`collect_queue_frames`/`collect_host_frames`) ahead of the header — no longer a bare 16-byte LSN sentinel. Original backlog note retained for provenance: real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs was already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. **Downstream: `config-upgrade-snapshot`'s former HARD dependency on this is now cleared** (`blocked_on: []`) — see that item above. **Also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0)** is now live: the standby-side snapshot-install/apply path is wired (`ha.c` ~3112-3147 wire-receive path task 6.1; `persist.c` ~4277-4285 local boot recovery task 8.1), with a decision path that requests a snapshot transfer when a standby's cursor falls below the leader's oldest retained WAL LSN (`ha.c:4413`). The bug-1147 WAL-trim cursor-bound guard (see Hardening below) is implemented in production, no longer just a stopgap.
|
||||
|
||||
## Hardening / Direct-Implementation Items
|
||||
|
||||
Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.
|
||||
|
||||
- [ ] **investigate: job_ctrl direct-mutate path (admin suspend/resume) — migrate to commit-then-apply?** — **recorded 2026-07-13, surfaced during job-array-single-wal-record's `persist_job_hook_fun` investigation.** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes) — confirmed the ONLY remaining live call site that still mutates state directly, THEN fires the `AFTER_EVENT` hook, with `persist_job_hook_fun` doing the real WAL append + async commit from inside the hook (not through `apply_committed_entry()`). This is the same mutate-before-commit shape as the bug-795/828/851/1060 divergence family raft-apply-decoupling was built to eliminate for job submit + worker facts, narrowly scoped to admin suspend/resume. Not yet known to have caused a live incident — rare, synchronous, admin-only traffic. **Investigate before committing to a fix**: (a) actual blast radius if a commit fails/times out post-mutation (does it ever get observed/notified before commit — violates criterion 2.4-style provisional-state-export concerns?); (b) live-apply vs replay-apply asymmetry for this one WAL entry type — does replay actually diverge from live in any reachable scenario, or is it benign in practice; (c) cost of migrating vs. accepted-residual status quo. Promote to a spec (likely a small `raft-apply-decoupling` follow-up, not standalone) only if the investigation finds real risk, not proactively.
|
||||
- [ ] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — `trim_wal_segments` (`persist.c:1374`) bounds deletion ONLY by the open segment (`if (seg_start_lsn < active_seg_lsn) unlink()`) — it consults **no** standby `acked_lsn`, **no** `committed_lsn`, **no** in-flight snapshot boundary. With no working snapshot backfill (`write_snapshot` emits a 16-byte sentinel only), a standby lagging by more than ~one open segment hits a log range that exists nowhere → permanent committed-prefix gap (the bug-795/828 family). This is **strategy-A retention discipline that should exist regardless**. Stopgap fix: compute `floor = min(all standby acked_lsn, in-flight snapshot boundaries)` under `p->lock` and refuse to delete at/above `floor`. **Folded into `raft-drive-async` task 3.2** (which moves checkpoint/trim onto a concurrent lane, worsening the race) — but stands alone if that slips. Does NOT fix empty/far-behind join (cursor=0); that needs `entity-bearing snapshots` (strategy B, backlog). Related: bug-795, bug-828, bug-1060.
|
||||
- [x] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — **IMPLEMENTED** as part of `raft-drive-async` task 3.2 (`raft-drive-async` is IMPLEMENTED, validate-impl GO 2026-07-18; see above). `trim_wal_segments` (`persist.c` ~2597-2644) now computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` — folding in active-standby `acked_lsn` and disconnected-slot `last_disconnected_lsn` — and only unlinks segments below that floor; the code comment at `persist.c:2634` explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that `entity-bearing snapshots` (strategy B) has landed as `ha-entity-snapshots` (see above), with a live decision path (`ha.c:4413`, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060.
|
||||
- [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_
|
||||
- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`.
|
||||
- [ ] validate_host() real authorization — currently a stub returning success for all peers; MUST be implemented before `CONF_ENABLE_UPGRADE_CONFIG=1`. Minimum: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` **and `OPCode_schedAdmin`** (added 2026-08-02, `scheduler-core`) with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.**
|
||||
- [ ] validate_host() real authorization — currently a stub returning success for all peers (`join.c:13-16`, `return 1;` unconditionally). **`CONF_ENABLE_UPGRADE_CONFIG` is already `1` in the standard build** (`config_repl.h:27-28`, no build-system override found) — the precondition this item was originally framed to guard against has already been crossed; `OP_upgradeConfig` (and `OPCode_schedAdmin`, added 2026-08-02, `scheduler-core`) are live and unauthenticated TODAY, not a future risk to prevent. Minimum fix: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` and `OPCode_schedAdmin` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.**
|
||||
- [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ **Part of the broader `peer-authentication-hardening` item above.**
|
||||
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
|
||||
- [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_
|
||||
- [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
|
||||
- [ ] xce_test WAL/snapshot inspection tooling — extend the offline inspector family in `xce_test` (sibling of the shipped `waldump`, `src/controller/testsuite/waldump.{c,h}` + `cmd_waldump`). Two remaining commands: **(a) `snapdump <snap_dir|snap_file>`** — decode the `XCESNAP\0` header (8-byte LSN; `persist.c:1266` `load_snapshot`) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (`.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md`): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. **(b) `walcompare <dirA> <dirB>`** — diff two nodes' WALs by `lsn → crc/jobid` to flag divergence/prefix-gaps between members instantly. Reuse the `waldump.c` reader/decoder; same zero-alloc capnp jobid decode + `c_capnproto_dep`/`zlib_dep` wiring already added to `xce_test_deps`. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. _(recorded 2026-06-20; `waldump` already done.)_
|
||||
- [ ] xce_test shell command support — add `!<command>` prefix to `xce_test` test scripts to execute arbitrary shell commands inline during test runs. Enables test scripts to drive environment setup/teardown (e.g., kill a node, write a file, run a helper) without calling out to a separate harness layer. Scope: parser change in the xce_test runner to detect `!`-prefixed lines, fork/exec the remainder as a shell command, fail the test on non-zero exit, capture stdout/stderr to the test log.
|
||||
- [→] scheduler framework enhancement — **PROMOTED to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/scheduler-framework/brief.md`). Original backlog seed superseded by the full brief with iterator-driven allocation architecture, configurable job picker, combinatorial parallel job allocation, and session-level unsatisfiable resreq cache. Scope narrowed from the original seed: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). Next: `/kiro-spec-requirements scheduler-framework`.
|
||||
- [x] scheduler framework enhancement — **PROMOTED to spec 2026-07-06, then DECOMPOSED 2026-07-27** into `scheduler-core` (**IMPLEMENTATION-COMPLETE**), `scheduler-allocation`, and `scheduler-preemption` — see **In Flight** above for current status. Scope notes from the original brief still apply: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). No longer tracked here.
|
||||
- [→] **perf-tuning-extraction** — **PROMOTED to spec 2026-07-20** (`clusterconfig-schema-split`, `.kiro/specs/clusterconfig-schema-split/`). Originally recorded 2026-07-06 from admin-config-tool discovery as a documentation/audit item; the audit (2026-07-20) found the fields are load-bearing enough to need real code changes, not just a doc pass — `candidates`/`vcpus`/hash-tuning fields (`jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`) are already dead weight in the `ClusterConfig` capnp blob (superseded by `--peers`/`--vcpus` CLI flags or never read into runtime state — see `src/docs/cluster-config-split-analysis.md`). **`port` is NOT dead** — a second review pass (2026-07-20) found `cluster_open_listener` (`cluster.c:1216-1258`) falls back to the blob's `port` field, and single-node mode (no `--peers` flag at all, `main.c:293-294`) depends on that fallback entirely; the spec now requires a replacement port-sourcing mechanism before the field can be removed (`clusterconfig-schema-split` Requirement 1.7), not a bare deletion. `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness, NOT the HA/Raft election heartbeat) were investigated and deliberately kept IN `ClusterConfig` for operational uniformity despite passing the independence rule — see `.kiro/steering/config-design.md`. `fetchJobInterval`/`netioTimeout` left unresolved, flagged out-of-scope in the new spec for a future pass.
|
||||
- [ ] **Raft/HA heartbeat interval+timeout as CLI parameters** — **NEW 2026-07-20 from clusterconfig-schema-split discovery.** The controller-to-controller HA/Raft election heartbeat (`ha->config.heartbeat_interval_ms`/`heartbeat_timeout_ms` — the one where a cross-node mismatch risks spurious failover) is 100% hardcoded today (`HA_DEFAULT_HEARTBEAT_INTERVAL_MS`/`HA_DEFAULT_HEARTBEAT_TIMEOUT_MS`, `cluster.c:1131`) — there is no `ClusterConfig` field for it, and per user decision (2026-07-20) there should never be one: these values must be available to a controller BEFORE it has validated/received cluster configuration (a controller needs its own election-timing parameters to even participate in the election that delivers config in the first place — a bootstrap/chicken-egg constraint, not an independence/frequency judgment call). Target: `--heartbeat-interval-ms`/`--heartbeat-timeout-ms` CLI flags following the exact `--vcpus` pattern (`OPT_*` + `struct arguments` + `myconf_t` field, `0` sentinel = use hardcoded default). Kept hardcoded for now; this is a future item, not blocking `clusterconfig-schema-split`.
|
||||
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ hostsGroup.POST("/control", RequirePermission("host:control"), handler)
|
|||
|
||||
## Permission String Format
|
||||
|
||||
`"action:scope"` — e.g., `"job:submit"`, `"host:view"`, `"queue:manage"`.
|
||||
`"action:scope"` — e.g., `"queue:submit"`, `"host:view"`, `"queue:control"`.
|
||||
|
||||
Scopes: `all` (unrestricted), `self` (own resources), `cluster`, `hostgroup`, `self_ug`, `self_hg`.
|
||||
|
||||
|
|
@ -39,7 +39,7 @@ Token secret is loaded from `config/auth.yaml` at startup. It must never appear
|
|||
|
||||
## Auth Backends
|
||||
|
||||
Backend selection is controlled by `config/auth.yaml`. Supported: `configfile`, `ldap`, `oauth2`, `pam`. All backends implement the same interface — adding a new backend means implementing that interface, not modifying middleware.
|
||||
Backend selection is controlled by `config/auth.yaml`'s `provider:` field, matched literally against a `switch` in `backend/auth/init.go` — an unrecognized value (e.g. `configfile` without the underscore) fails apiserver startup with "unsupported auth provider". Valid identifiers: `config_file`, `pam`, `ldap`, `oauth2`. Only `config_file` (dev mode, users defined in the YAML) and `pam` (Linux + CGO only) are functional today — `ldap` and `oauth2` are unconditional stubs whose `Authenticate()` always returns "not yet implemented". All backends implement the same interface — adding a new backend means implementing that interface, not modifying middleware.
|
||||
|
||||
## Input Validation
|
||||
|
||||
|
|
@ -47,14 +47,14 @@ Validate at the API boundary inside `preCheck{Resource}Request()` helpers — be
|
|||
|
||||
## Secrets
|
||||
|
||||
- JWT secret key: `config/auth.yaml` (file-system protected, chmod 0600)
|
||||
- JWT secret key: `config/auth.yaml` — should be file-system protected (e.g. chmod 0600); this is not currently enforced by the code or deploy tooling, so treat it as an operator responsibility, not a guarantee
|
||||
- LDAP bind credentials: `config/auth.yaml`
|
||||
- **Never log** tokens, passwords, secret keys, or full request bodies containing credentials
|
||||
- Validate all required config at startup; fail fast with a clear error if missing
|
||||
|
||||
## Rate Limiting and IP Whitelist
|
||||
|
||||
Rate limiting (`middleware/rate_limit.go`) and IP whitelist (`middleware/ip_whitelist.go`) are applied before JWT validation in the middleware chain. They use `ctx.ClientIP()` and abort with 429 / 403 respectively. Both are configured in `config/` YAML — do not hardcode thresholds.
|
||||
Rate limiting (`middleware/rate_limit.go`) and IP whitelist (`middleware/whitelist.go`, which implements both `WhitelistURLMiddleware()` and `WhitelistIPMiddleware()`) are applied before JWT validation in the middleware chain. They use `ctx.ClientIP()` and abort with 429 / 403 respectively. Both are configured in `config/` YAML — do not hardcode thresholds.
|
||||
|
||||
## Transport
|
||||
|
||||
|
|
@ -64,6 +64,6 @@ XCE uses **server-side TLS only** — encryption without peer certificate verifi
|
|||
|
||||
**Clients talking to the API server** (xcecli, or any HTTP client of `apiserver`) must support **both `http://` and `https://`** — the scheme is a per-deployment/user choice (`APIServerInfo.URL`), not something the client hardcodes. When the scheme is `https://`, set `InsecureSkipVerify: true` on the TLS client config (see `xcecli/common/request/request.go`'s `NewAPIClient`) rather than attempting cert validation — the API server's cert is typically self-signed/auto-generated and the client has no CA to validate it against.
|
||||
|
||||
**Clients talking to an XCE controller directly** (worker, scheduler, and any other libxce-based client via `cluster.c`'s `do_connect()`) need **only server-side certs — no client-side certs are required or supported**. `xce_cluster_config_t` has no `ca_cert_path`/client-cert fields; do not add any. TLS verification mode is `SSL_VERIFY_NONE` throughout — this matches the API-server posture above, just at the C/libxce layer instead of Go/HTTP.
|
||||
**Clients talking to an XCE controller directly** (worker, scheduler, and any other libxce-based client) need **only server-side certs — no client-side certs are required or supported**. The live connection path is `xcecd_join_cluster()` (`src/worker/src/controller_connector.c`) calling `xcecli_start`/`xcecli_join` (`src/lib/client.c`), which drives `endpoint_connect()` in `src/lib/netio.c` over an `SSL_CTX` built by `create_ssl_context(NULL, NULL)` (`src/lib/xcectx.c`) — note that `cluster.c`'s `do_connect()`/`xce_connect_cluster()`/`xce_reconnect_init`/`xce_reconnect_step` are an unused stub path (explicitly documented as "to be wired in a later task") with no callers outside `cluster.c` and its own testsuite; don't point at them as the live mechanism. `xce_cluster_config_t` has no `ca_cert_path`/client-cert fields; do not add any. TLS verification mode is `SSL_VERIFY_NONE` throughout — this matches the API-server posture above, just at the C/libxce layer instead of Go/HTTP.
|
||||
|
||||
**Known follow-up (not yet scoped):** a HAProxy reverse proxy in front of `apiserver` is planned to terminate HTTPS, likely with a CA-trusted cert (Let's Encrypt candidate, unconfirmed), and whether `apiserver`'s own `RunTLS`/`httpsConfig.Enabled` listener stays in the picture is also undecided. Once both are settled, revisit xcecli's blanket `InsecureSkipVerify: true`: if the client-facing endpoint ends up CA-trusted, default to real verification (Go's default `tls.Config{}` validates against the system trust store with no extra plumbing) and make skip-verify an explicit opt-in for self-signed deployments only — matching the automated security review's suggested fix. Don't implement this speculatively; the right shape depends on which of those two decisions lands where.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
## Organization Philosophy
|
||||
|
||||
**Layered monorepo, component-per-directory.** Each major subsystem lives under `src/<component>/` with its own Meson build file and CLAUDE.md. The Go frontend is isolated in `src/frontend/` as two independent Go modules. All C daemons share `src/lib/` (libxce) and public headers from `src/include/xce/`.
|
||||
**Layered monorepo, component-per-directory.** Each major subsystem lives under `src/<component>/` with its own Meson build file and CLAUDE.md. The Go frontend is isolated in `src/frontend/` as two independent Go modules. All C daemons share `src/lib/` (libxce) and public headers from `src/include/`.
|
||||
|
||||
```
|
||||
src/
|
||||
|
|
@ -14,14 +14,16 @@ src/
|
|||
├── scheduler/ C — job scheduling engine (plugin architecture)
|
||||
├── autoscaler/ C — autoscaling engine (provider plugins)
|
||||
├── frontend/ Go — apiserver (Gin REST) + xcecli (Cobra CLI)
|
||||
├── include/xce/ C — public shared headers consumed across all daemons
|
||||
├── xceadmin/ Python — admin CLI tool
|
||||
├── include/ C — public shared headers consumed across all daemons
|
||||
│ └── xce/ C — subset of public headers under the xce/ prefix
|
||||
├── schema/ Cap'n Proto schema files (c.capnp, controller.capnp)
|
||||
└── docs/ Project-wide documentation (markdown)
|
||||
```
|
||||
|
||||
## C Daemon Layout Pattern
|
||||
|
||||
Each C daemon (`controller/`, `worker/`, `scheduler/`, `autoscaler/`) follows:
|
||||
`worker/` and `scheduler/` follow:
|
||||
|
||||
```
|
||||
src/<component>/
|
||||
|
|
@ -32,7 +34,10 @@ src/<component>/
|
|||
testsuite/ — test programs (test_*.c)
|
||||
```
|
||||
|
||||
**Exception**: `controller/` keeps its `.c` files flat in the component root (not in a nested `src/`). All other C daemons use `src/<component>/src/`.
|
||||
**Exceptions**:
|
||||
- `controller/` keeps its `.c` files flat in the component root (not in a nested `src/`), and does have its own `testsuite/`.
|
||||
- `worker/` has no `testsuite/` — no `test_*.c` files exist anywhere in the component.
|
||||
- `autoscaler/` does not follow this pattern at all: it has no CLAUDE.md, and its daemon code lives two levels deeper in `src/autoscaler/scaler/` (flat `.c` files plus its own `include/` and `testsuite/`), alongside `src/autoscaler/providers/`.
|
||||
|
||||
Plugin-based components add their own subdirectory:
|
||||
- `scheduler/` — plugins in `src/plugins/`
|
||||
|
|
@ -62,7 +67,7 @@ Plugin-based components add their own subdirectory:
|
|||
### C
|
||||
|
||||
- **Files**: `snake_case.c` / `snake_case.h` — one domain per file
|
||||
- **Public headers**: `src/include/xce/<name>.h` — the libxce API boundary
|
||||
- **Public headers**: `src/include/<name>.h` (e.g. `persist.h`, `netio.h`, `resreq.h`) — the libxce API boundary; a small subset (`client_failover.h`, `cluster.h`, `cluster_def.h`, `errnox.h`, `types.h`) lives under `src/include/xce/` and is included as `xce/<name>.h`
|
||||
- **Component headers**: `src/<component>/include/<name>.h`
|
||||
- **Test files**: `test_<unit>.c` (e.g., `test_ha_cluster.c`, `test_persist_wal.c`)
|
||||
- **Functions**: `snake_case` with component prefix — `ha_*`, `cluster_*`, `xcecd_*`, `xce_*`
|
||||
|
|
@ -74,8 +79,8 @@ Plugin-based components add their own subdirectory:
|
|||
|
||||
- **Files**: one file per resource — `jobs.go`, `queues.go`, `hosts.go`
|
||||
- **Tests**: `*_test.go` co-located; integration tests as `integration_test.go`, `<resource>_control_test.go`
|
||||
- **Controller structs**: `{Resource}Controller`
|
||||
- **Handler methods**: `{Action}{Resource}` (e.g., `SubmitJob`, `ListJobs`)
|
||||
- **Controller structs**: `{Resource}Controller` (e.g., `JobController`, `ClusterController`, `QueueController`); one exception — `hosts.go` uses `HostHandler`
|
||||
- **Handler methods**: `{Action}{Resource}` (e.g., `CreateJob`, `GetJobsInfo`, `ControlJobByID`)
|
||||
- **Permission strings**: `"action:scope"` (e.g., `"job:view:self"`)
|
||||
- **JSON tags**: `snake_case`
|
||||
|
||||
|
|
|
|||
|
|
@ -8,9 +8,9 @@ xcecli (Go/Cobra) ──REST──► apiserver (Go/Gin) ──CGO──► C da
|
|||
↑↑↑ all linked against libxce (src/lib/)
|
||||
```
|
||||
|
||||
Request path inside apiserver: `Gin Router → Middleware (CORS/RateLimit/IPWhitelist/JWT) → RBAC Check → Controller → Services → In-Memory Store`
|
||||
Request path inside apiserver: `Gin Router → Middleware (WhitelistURL → WhitelistIP → RateLimit → JWT → CORS) → RBAC Check → Controller → Services → In-Memory Store`
|
||||
|
||||
All controller state is in-memory, seeded at startup from YAML config. No persistent database in the API tier. C daemons communicate via TCP using a custom messaging layer: iomgr (epoll-based I/O loop), vtaskmgr (thread pool for packet dispatch), and xce_request/xce_reply for non-blocking send/receive. WAL + checkpoint provide persistence at the controller level.
|
||||
All controller state is in-memory. No persistent database in the API tier; job/queue/host state is pushed live into the in-memory stores from the C controller backend over CGO (see Key Technical Decision #2), not loaded from a file. C daemons communicate via TCP using a custom messaging layer: iomgr (epoll-based I/O loop), vtaskmgr (thread pool for packet dispatch), and xce_request/xce_reply for non-blocking send/receive. WAL + checkpoint provide persistence at the controller level.
|
||||
|
||||
## Core Technologies
|
||||
|
||||
|
|
@ -52,18 +52,20 @@ nix build .#xcecli # CLI only
|
|||
### Controller Pattern (Go)
|
||||
|
||||
```go
|
||||
// Struct: {Resource}Controller
|
||||
type JobController struct { ... }
|
||||
// Struct: {Resource}Controller (typically an empty struct{})
|
||||
type JobController struct{}
|
||||
|
||||
// Handler: {Action}{Resource}
|
||||
func (c JobController) SubmitJob(ctx *gin.Context) {
|
||||
func (j JobController) CreateJob(ctx *gin.Context) {
|
||||
// Validate input
|
||||
req, err := preCheckSubmitJobRequest(ctx)
|
||||
if err != nil { ReturnError(ctx, code, err, nil); return }
|
||||
var req models.JobSubmitRequest
|
||||
if err := utils.BindJSONStrict(ctx, &req); err != nil { ReturnError(ctx, code, err.Error(), nil); return }
|
||||
parsedIndices, err := preCheckJobRequest(&req)
|
||||
if err != nil { ReturnError(ctx, code, err.Error(), nil); return }
|
||||
|
||||
// Business logic
|
||||
result, err := c.svc.Submit(req)
|
||||
if err != nil { ReturnError(ctx, code, err, nil); return }
|
||||
// Business logic — package-level services function, not a controller field
|
||||
err = services.SubmitJob(&newJob, arraySize)
|
||||
if err != nil { ReturnError(ctx, code, err.Error(), nil); return }
|
||||
|
||||
ReturnSuccess(ctx, code, "submitted", result)
|
||||
}
|
||||
|
|
@ -86,15 +88,15 @@ Early return on every error. Log with zap fields before returning.
|
|||
|
||||
### Auth and RBAC
|
||||
|
||||
- Auth backend set in `config/auth.yaml`. Supports: configfile, LDAP, OAuth2, PAM.
|
||||
- Permissions as `"action:scope"` strings: `"job:view:self"`, `"queue:submit:all"`.
|
||||
- Auth backend set in `config/auth.yaml` (`provider: config_file | pam | ldap | oauth2`). Only `config_file` and `pam` (Linux + CGO) are functional; `ldap` and `oauth2` are documented as future/stub providers whose `Authenticate()` always returns a "not yet implemented" error.
|
||||
- Permissions are a plain action string (e.g. `"job:view"`, `"queue:submit"`) paired with a separate structured `domain` list (e.g. `{type: self}`, `{type: all}`) in `rbac.yaml` — not a combined `"action:scope"` string. `controllers.RequirePermission()` checks the action string only; domain/scope is evaluated separately.
|
||||
- RBAC loaded from `config/rbac.yaml`, enforced by `controllers.RequirePermission()` middleware.
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
1. **No Go workspace** — `apiserver` and `xcecli` are independent modules; changes to shared types require updating both.
|
||||
2. **In-memory API state** — no database in Go tier; state loaded from YAML at startup via `store.LoadBuiltinData()`.
|
||||
2. **In-memory API state** — no database in Go tier; `store.InitStores()` allocates empty per-resource stores at startup, and job/queue/host/etc. state is then pushed in live from the C controller backend over CGO (`backend.RegisterToBackend()` + `backend.StartNotificationListener()` in `cmd/main.go`). Only RBAC (`config/rbac.yaml`) and auth (`config/auth.yaml`) are actually loaded from YAML at startup.
|
||||
3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell.
|
||||
4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin.
|
||||
5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output.
|
||||
6. **Single-writer HA** — controller is always one active primary; standby election via Raft-style consensus (term-based voting, freshness gates) in the `src/lib/raft/` shared module (`raft.c`/`raft.h`), with the single-writer constraint avoiding Raft log replication complexity.
|
||||
6. **Single-writer HA with full Raft log replication** — controller is always one active primary; standby election and log replication use a Raft protocol (term-based voting, freshness gates, `AppendEntries` request/reply, per-peer `next_index[]`/`match_index[]` progress vectors, `(term,index)` log matching/truncation, commit-index advance) implemented in the `src/lib/raft/` shared module (`raft.c`/`raft.h`) and validated by the `raft-log-replication`, `raft-commit-safety`, `raft-ha-core-safety`, and `ha-term-safety` specs.
|
||||
|
|
|
|||
|
|
@ -48,10 +48,12 @@ func TestPermissions(t *testing.T) {
|
|||
- `make test-backend` — CGO-backed backend tests (requires nix dev shell + build dir)
|
||||
- `make test-integration` — full regression suite (requires `XCE_BIN` and `SRV_URL` env vars)
|
||||
|
||||
## C Tests (controller / worker / scheduler / lib)
|
||||
## C Tests (controller / scheduler / lib)
|
||||
|
||||
**Location**: `testsuite/` directory inside each C component.
|
||||
|
||||
**Note**: `src/worker` has no `testsuite/` directory and no meson `test()` registrations today — it currently carries no C test coverage of its own.
|
||||
|
||||
**Structure**: one `.c` test program per unit:
|
||||
|
||||
```
|
||||
|
|
@ -83,7 +85,7 @@ extern void reset_ha_state(ha_state_t *s);
|
|||
|
||||
- Go: 80% minimum on business logic (controllers, services, store)
|
||||
- C: every new function in a daemon must have at least one test case in `testsuite/`
|
||||
- Integration: all public REST endpoints must have at least one integration test case in `regression/`
|
||||
- Integration: all public REST endpoints must have at least one integration test case in `src/frontend/tests/regression/`
|
||||
|
||||
## What Not to Mock
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue