merge-fix: drop Rongsong bookkeeping missed in merge (.kiro/ reports/ CHANGES.md)
First git rm during merge was refused for worktree-modified files; only .wolf/ got removed. Clean up the remaining OpenWolf/spec bookkeeping and weekly reports so the merged branch carries source fixes only.
This commit is contained in:
lei.chen
parent
3b6f53bb65
commit
3dbb96d4e6
343 changed files with 0 additions and 56290 deletions
|
|
@ -1,145 +0,0 @@
|
|||
# API Standards
|
||||
|
||||
## Philosophy
|
||||
|
||||
Two API surfaces: Go REST API (apiserver) and C daemon messaging (Cap'n Proto over TCP). Both follow a consistent pattern: validate at the boundary, delegate business logic, return structured responses. Non-blocking I/O throughout the daemon layer.
|
||||
|
||||
## Go REST API (apiserver)
|
||||
|
||||
### Handler Pattern
|
||||
|
||||
Every resource follows a single consistent pattern:
|
||||
|
||||
```go
|
||||
// {Resource}Controller struct
|
||||
type JobController struct{}
|
||||
|
||||
// {Action}{Resource} handler method
|
||||
func (j JobController) SubmitJob(ctx *gin.Context) {
|
||||
// 1. Validate input at the boundary
|
||||
req, err := preCheckSubmitJobRequest(ctx)
|
||||
if err != nil {
|
||||
ReturnError(ctx, common.INVALID_PARAMS_CODE, err.Error(), nil)
|
||||
return
|
||||
}
|
||||
|
||||
// 2. Delegate to service layer
|
||||
result, err := j.svc.Submit(req)
|
||||
if err != nil {
|
||||
ReturnError(ctx, common.ERROR_CODE, err.Error(), nil)
|
||||
return
|
||||
}
|
||||
|
||||
// 3. Return structured success
|
||||
ReturnSuccess(ctx, common.SUCCESS_CODE, "submitted", result)
|
||||
}
|
||||
```
|
||||
|
||||
**Rules**:
|
||||
- Early return on every error — `ReturnError(...); return`
|
||||
- Keep `preCheck*` functions in the same file as the handler
|
||||
- Controllers delegate to services — business logic never lives in handlers
|
||||
- Dependencies flow inward: `controllers → services → store/backend`
|
||||
|
||||
### Response Format
|
||||
|
||||
Every HTTP response uses a consistent envelope via two helpers in `controllers/response.go`:
|
||||
|
||||
```go
|
||||
// Success (always HTTP 200)
|
||||
ReturnSuccess(ctx, code, msg, data)
|
||||
// → {code: 200, msg: "success", data: {...}}
|
||||
|
||||
// Error (HTTP status == code parameter)
|
||||
ReturnError(ctx, code, msg, details)
|
||||
// → {code: 401, msg: "access forbidden", details: nil}
|
||||
```
|
||||
|
||||
### Standard Error Codes
|
||||
|
||||
All codes defined in `common/common.go`:
|
||||
|
||||
| Code | Constant | Usage |
|
||||
|------|----------|-------|
|
||||
| 200 | `SUCCESS_CODE` | All success responses |
|
||||
| 400 | `INVALID_PARAMS_CODE` | Malformed/missing input |
|
||||
| 401 | `ERROR_AUTH_CHECK_TOKEN_CODE` | Missing/invalid JWT |
|
||||
| 403 | `ERROR_AUTH_CODE` | Valid JWT, insufficient permission |
|
||||
| 404 | `ERROR_NOT_FOUND_CODE` | Resource not found |
|
||||
| 429 | `ERROR_TOO_MANY_REQUESTS_CODE` | Rate limit exceeded |
|
||||
| 500 | `ERROR_CODE` | Unexpected server error |
|
||||
|
||||
### Middleware Chain
|
||||
|
||||
Order is defined in `router/routers.go` and is **non-negotiable**:
|
||||
|
||||
```
|
||||
WhitelistURL → WhitelistIP → RateLimit → ValidateHandler (JWT) → CORS → RBAC
|
||||
```
|
||||
|
||||
- **WhitelistURL**: filter by allowed paths (applied first for efficiency)
|
||||
- **WhitelistIP**: abort 403 if client IP not in whitelist
|
||||
- **RateLimit**: 1000 req/sec, burst 2000 (global config, `config/app.yaml`)
|
||||
- **ValidateHandler**: parse and validate JWT from `Authorization: Bearer` header; validate body content-type
|
||||
- **CORS**: origin checks via config
|
||||
- **RBAC**: `RequirePermission("action:scope")` per-route, enforced by `controllers/rbac_check.go`
|
||||
|
||||
### Permission Format
|
||||
|
||||
`"action:scope"` — scopes: `all`, `self`, `cluster`, `hostgroup`, `self_ug`, `self_hg`.
|
||||
Defined in `config/rbac.yaml`. Roles support inheritance via `includes:`.
|
||||
|
||||
## C Daemon Messaging
|
||||
|
||||
### Cap'n Proto Wire Format
|
||||
|
||||
All inter-daemon messages are Cap'n Proto serialized. Schema: `src/schema/controller.capnp`.
|
||||
|
||||
**Request** (`xce_request_t`):
|
||||
- `opcode` — OPCode enum (43 values)
|
||||
- `seq` — request sequence number
|
||||
- `from` — sender peer info (host/key)
|
||||
- `body` — union of 29 message types (including the empty `nomsg` placeholder)
|
||||
|
||||
**Reply** (`xce_reply_t`):
|
||||
- `opcode` — echoes the originating request's opcode by default (`init_reply_by_req()` sets `reply->opcode = req->opcode`); many call sites override it with a type-specific opcode (e.g. `OPCode_preVoteReply`, `OPCode_walAck`, `OPCode_schedAdmin`). Consumers dispatch on this value, so it is not a fixed `OPCode_reply`/`OPCode_redirect` pair — that pairing only applies to the separate wrapper `xce_request_t` that `xce_reply()` builds internally.
|
||||
- `status` — error code (0 = success)
|
||||
- `seq` — echoes request seq
|
||||
- `data` — union of reply payloads (clusterconf, jobid, stateReport, walackreplymsg, etc.)
|
||||
|
||||
### Key OPCodes
|
||||
|
||||
| Group | OPCodes | Purpose |
|
||||
|-------|---------|---------|
|
||||
| Control | `jobsub (11)`, `jobdispatch (8)`, `jobcontrol (15)` | Job lifecycle |
|
||||
| Cluster | `hello (2)`, `join (3)`, `clusterconf (17)`, `heartbeat (25)` | Membership and liveness |
|
||||
| Replication | `replConfig (29)`, `replConfigPush (30)`, `walCatchupEntry (37)`, `snapshotTransfer (36)` | Config + WAL replication |
|
||||
| Vote | `requestVote`, reply | Raft election |
|
||||
| Data | `push (6)`, `get (5)`, `notify (10)` | Data exchange |
|
||||
|
||||
### Daemon I/O Functions
|
||||
|
||||
```c
|
||||
// Enqueue request to endpoint send buffer (non-blocking, always)
|
||||
int xce_request(endpoint_t *ep, xce_request_t *req);
|
||||
|
||||
// Enqueue reply to inbound request sender (non-blocking)
|
||||
int xce_reply(endpoint_t *ep, xce_peerinfo_t *host, xce_reply_t *reply);
|
||||
|
||||
// Cap'n Proto serialize request to packet
|
||||
int xce_encode(xce_request_t *req, io_packet_t *pkt);
|
||||
|
||||
// Cap'n Proto deserialize packet to request
|
||||
int xce_decode(io_packet_t *pkt, xce_request_t *req);
|
||||
```
|
||||
|
||||
**msg_handler callback**: Each endpoint has `ep->msg_handler` — a function pointer `int (*)(void*)` called by iomgr when a packet arrives. Handlers must return quickly (vtask context). Registered per-endpoint during connection setup.
|
||||
|
||||
### Adding a New Opcode
|
||||
|
||||
1. Add the opcode constant to the `OPCode` enum in `src/schema/controller.capnp`
|
||||
2. Add the message struct to `src/schema/controller.capnp` (Request/Reply union arms)
|
||||
3. Rebuild capnp schemas (automatic via meson)
|
||||
4. Add a `case OPCode_...:` in the `switch (request.opcode)` dispatch inside `myreq_handler()` in `cluster.c` (this runs after `ha_dispatch_gate`, which only applies a role-based allow/deny filter, not per-opcode handler logic)
|
||||
5. Add encode/decode paths in `src/lib/request.c` if using raw payloads
|
||||
6. **If the new opcode is primary-only** (absent from `ha_dispatch_gate`'s `CTRL_STANDBY` allow-list), confirm its rejection reply reaches a client that can actually recover — `myreq_handler`'s generic gate-rejection path (`cluster.c`, right after the `ha_dispatch_gate()` call) must populate the redirect fields via `fill_redirect_reply()` (`join.c`) when rejecting for `-XCE_ERR_STANDBY`, mirroring `OPCode_join`'s handler. Every client-side redirect-follower in this codebase (`xceadmin.discovery.follow_redirects()`, libxce's own failover path) is already generic/opcode-agnostic and works automatically the moment the server supplies a redirect — don't assume "the client should just retry" is a client-side gap without first checking whether the server ever gave it anything to retry against (bug-1700: the identical gap was independently found and documented for `OPCode_upgradeConfig` first — `test_e2e_live_ha_push.py`'s module docstring — then shipped again under `OPCode_schedAdmin` before either was actually fixed).
|
||||
|
|
@ -1,163 +0,0 @@
|
|||
# Build & Deployment
|
||||
|
||||
## Philosophy
|
||||
|
||||
A single Heremetic build system (Meson + Nix flakes) produces all artifacts — C daemons, Go binaries, AppImage. Only the `asan` profile (the default for `/xce-build`) includes both debug symbols and ASAN; `debug` builds symbols without ASAN, and `release` builds (including the nix `xcecore` production package) carry neither. Tests run as part of the build, not after. Deployment for HA testing uses a scripted local cluster with per-node data directories and cert management.
|
||||
|
||||
## Build Commands
|
||||
|
||||
All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
|
||||
|
||||
```bash
|
||||
/xce-build # all components + all tests
|
||||
/xce-build lib # libxce + lib tests
|
||||
/xce-build controller # lib + controller + controller tests + xce_test driver
|
||||
/xce-build scheduler # lib + scheduler + scheduler tests
|
||||
/xce-build worker # lib + worker (daemon only, no unit tests)
|
||||
/xce-build apiserver # lib + apiserver + xcecli + Go unit tests
|
||||
/xce-build xcecli # xcecli + Go tests
|
||||
/xce-build xceadmin # xceadmin (meson-driven; builds src/xceadmin's venv/tests + an installable wheel)
|
||||
```
|
||||
|
||||
`xceadmin` ignores the profile arg (always the same pinned, non-editable, out-of-tree install + test, regardless of `asan`/`debug`/`release`). Beyond the build-dir venv, it also produces a real wheel at `<build_dir>/xceadmin/dist/xceadmin-<version>-py3-none-any.whl` (built via `pip wheel` from the same schema-synced tree the tests just ran against, only after they pass) — `pip install` it anywhere a compatible Python is available, no nix/meson needed at install time. `pyproject.toml` declares all 4 real runtime deps (`click`, `pycapnp`, `PyYAML`, `jsonschema` — verified against every actual `import` in `src/xceadmin`/`src/xcecommon`); the one caveat is `pycapnp`, whose PyPI-prebuilt wheels are usually fine outside the nix shell but aren't the same guaranteed-consistent link against the capnproto C++ runtime `python3Packages.pycapnp` gets.
|
||||
|
||||
`controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it.
|
||||
|
||||
Each invocation:
|
||||
1. `nix develop` into the hermetic shell
|
||||
2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build — `rm -rf <dir>` then reconfigure, unconditionally, for EVERY component including single-component invocations)
|
||||
3. `meson compile -C <dir> [targets...]` — per-component invocations pass an explicit target list scoped to just that component (e.g. `controller` compiles only `xce_controller controller_unit_test controller/xce_test`); only `/xce-build all` compiles with no target args, building everything
|
||||
4. Run registered meson tests (skipped for non-`asan` profiles — see below)
|
||||
|
||||
**Consequence (bug-1565) and fix:** because step 2 wipes the whole build dir but step 3 only rebuilds the requested component, running `/xce-build xceadmin` (or any other single component) after `/xce-build controller` in the SAME build dir silently deletes the already-built `xce_controller`/`xce_test`/etc — they're gone until something explicitly rebuilds them too. Two ways to get multiple binaries from the same dir together: run `/xce-build all` once instead of separate single-component invocations, **or** pass `--incremental` to a single-component invocation (`/xce-build xceadmin --incremental`) to reconfigure the existing dir in place (`meson setup --reconfigure`) instead of wiping it — preserves whatever else was already built there. `--incremental` is opt-in, not the default: this project's capnp-generation `custom_target` has documented, unfixed missing-dependency-edge races (buglog.json, "retry once before debugging") that a full wipe papers over as a loud, retry-fixable clean-build failure; skipping the wipe could instead let the same gap cause silent staleness (a schema edit not triggering regeneration, so the build reports success against stale generated code) — rebuild without the flag if anything looks wrong afterward.
|
||||
|
||||
**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.
|
||||
|
||||
## ASAN Integration
|
||||
|
||||
ASAN is enabled in every build by default (`-Db_sanitize=address`). Controller tests set `detect_leaks=0` (known pre-existing leaks flagged as technical debt). Test binaries print LeakSanitizer reports on stderr after exit.
|
||||
|
||||
## Meson Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── meson.build # top-level: subdir() order, project()
|
||||
├── lib/meson.build # libxce shared library + libxce_test
|
||||
├── controller/meson.build # xce_controller daemon + controller_unit_test
|
||||
├── worker/meson.build # xcecd daemon
|
||||
├── scheduler/meson.build # scheduler daemon + test
|
||||
├── autoscaler/meson.build # libscaler static lib + provider tools (no daemon binary)
|
||||
├── frontend/apiserver/meson.build # Go apiserver via tools/go_build
|
||||
├── frontend/xcecli/meson.build # Go xcecli via tools/go_build
|
||||
└── xceadmin/meson.build # xceadmin config-build tool (used by /xce-deploy-ha)
|
||||
```
|
||||
|
||||
**Dependency flow**: `lib → {controller, worker, scheduler, apiserver}`
|
||||
|
||||
**Test registration**:
|
||||
```python
|
||||
# controller/meson.build
|
||||
test('controller test', controller_unit_test,
|
||||
suite: 'controller',
|
||||
timeout: 300,
|
||||
env: ['ASAN_OPTIONS=detect_leaks=0'])
|
||||
```
|
||||
|
||||
**XCE_TESTING flag**: The controller_unit_test executable compiles with `-DXCE_TESTING` to enable test-only seams (hook variables, reset functions).
|
||||
|
||||
## Nix Flake
|
||||
|
||||
Located at `src/flake.nix`. Provides:
|
||||
|
||||
```bash
|
||||
nix develop # hermetic dev shell (meson, pkgconfig, capnp, packcc, Go 1.23.4, etc.)
|
||||
nix build .#xcecore # C daemons + apiserver (release)
|
||||
nix build .#xcecore_debug # default package: debug + ASAN
|
||||
nix build .#xcecli # CLI only
|
||||
```
|
||||
|
||||
**AppImage**: `nix build .#xcecore`/`.#xcecore_debug` do not produce an AppImage — they only install `xce` (a copy of `tools/appimage/entry.sh`, see `src/meson.build`) as a plain shell entry-point binary. The `xce` flake package (`appImageEntryPackage` in `src/flake.nix`) wraps that entry script with bundled daemons, haproxy, glow, and less; the actual `.AppImage` file is produced separately with `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` (see README.md, "3.5 build AppImage package").
|
||||
|
||||
## Local HA Cluster Deployment
|
||||
|
||||
`/xce-deploy-ha` (skill at `.claude/skills/xce-deploy-ha/`):
|
||||
|
||||
```bash
|
||||
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
|
||||
```
|
||||
|
||||
This creates under `/tmp/xce-deploy/`:
|
||||
```
|
||||
certs/ # TLS certs for all peers
|
||||
controller/
|
||||
ctrl-1/
|
||||
conf/{current, cluster.bin.<tag>} # primary's config (tag = sha256sum prefix)
|
||||
data/ # WAL + checkpoint storage
|
||||
ctrl-2/
|
||||
conf/ # receives config via OPCode_replConfigPush
|
||||
data/
|
||||
ctrl-3/ (same as ctrl-2)
|
||||
xceadmin-site/ # xceadmin/xcecommon extracted from the wheel (if built) -- standalone, see below
|
||||
logs/ # per-node + worker/scheduler log files
|
||||
scripts/
|
||||
start.sh ctrl-N # launch a single node
|
||||
start-worker.sh [name] # launch xcecd_virtual against the SAME --peers (if worker built)
|
||||
start-scheduler.sh # launch xce_scheduler against the SAME --peers (if scheduler built)
|
||||
xceadmin ... # standalone xceadmin wrapper (if the wheel was built), forwards to xceadmin-site
|
||||
cleanup.sh # wipe data/standby-conf/logs, keep ctrl-1 config + certs
|
||||
```
|
||||
|
||||
`start-worker.sh`/`start-scheduler.sh`/`xceadmin` are soft-skipped (warned, not a deploy failure) when the corresponding binary/wheel doesn't exist in the same build dir as the controller binary — run `/xce-build worker`/`/xce-build scheduler`/`/xce-build xceadmin` first if you need them. `scripts/xceadmin` is a genuinely standalone install: the wheel is extracted (not installed via a `--system-site-packages` venv — this nix devShell injects `click`/`pycapnp`/`PyYAML`/`jsonschema` via `$PYTHONPATH`, not the base interpreter's own site-packages, so a venv can't see them) and the wrapper bakes in the resolved `$PYTHONPATH` at deploy time, so it works with zero nix/network dependency at runtime — verified against a completely stripped environment.
|
||||
|
||||
**Startup** (from deploy dir, via the bash resolved in the deploy environment — see below):
|
||||
```bash
|
||||
BASH_BIN=$(nix develop ./src --command which bash) # never a bare `bash`; see below
|
||||
"$BASH_BIN" scripts/start.sh ctrl-1 -vvv & # primary first
|
||||
sleep 0.5
|
||||
"$BASH_BIN" scripts/start.sh ctrl-2 -vvv & # standby (0.5s stagger for election ordering)
|
||||
sleep 0.5
|
||||
"$BASH_BIN" scripts/start.sh ctrl-3 -vvv &
|
||||
|
||||
# Optional, once at least one controller is up:
|
||||
"$BASH_BIN" scripts/start-worker.sh worker-1 -vvv &
|
||||
"$BASH_BIN" scripts/start-scheduler.sh -v &
|
||||
```
|
||||
(`-vvv` is a positional arg to `start.sh`, not a `VERBOSE` env var — the generated script only forwards `$*`.)
|
||||
|
||||
**Why resolve `$BASH_BIN` explicitly**: every script this deploy generates is bash (arrays, `+=`, `<<<`), and `deploy-ha.sh`'s own shebang has been found locally altered to `#!/bin/sh` while still using bash-only syntax — it only kept working because this environment's `/bin/sh` happens to be symlinked to bash. Resolving `which bash` *inside* `nix develop` once and using it explicitly for every invocation (instead of a bare `bash` or trusting a file's shebang) sidesteps that class of mismatch entirely; `deploy-ha.sh` also bakes this same resolved path into the shebang of every script it generates.
|
||||
|
||||
**Clean restart** (keeps certs + ctrl-1 config, wipes all data):
|
||||
```bash
|
||||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
kill -9 $(pgrep -f xcecd_virtual) 2>/dev/null || true
|
||||
kill -9 $(pgrep xce_scheduler) 2>/dev/null || true
|
||||
"$BASH_BIN" scripts/cleanup.sh
|
||||
# Then re-start all nodes (and worker/scheduler, if you had them running)
|
||||
```
|
||||
|
||||
## Go Frontend Build
|
||||
|
||||
Go builds use `src/frontend/Makefile` for development, Meson for production:
|
||||
|
||||
```bash
|
||||
make build-server # → /tmp/apiserver
|
||||
make build-cli # → /tmp/xce_static (CGO_ENABLED=0)
|
||||
make test # unit tests (CGO_ENABLED=0)
|
||||
make test-backend # CGO-backed backend tests (needs nix dev shell)
|
||||
make test-integration # full regression suite (needs XCE_BIN + SRV_URL)
|
||||
```
|
||||
|
||||
Go modules are independent — `apiserver` and `xcecli` have separate `go.mod` files, no `go.work`.
|
||||
|
||||
## Production Artifacts
|
||||
|
||||
| Artifact | Build command | Output |
|
||||
|----------|-------------|--------|
|
||||
| Controller daemon | `nix build .#xcecore` | `xce_controller` binary |
|
||||
| Worker daemon | same | `xcecd` binary |
|
||||
| Scheduler daemon | same | `xce_scheduler` binary |
|
||||
| REST API server | same | `xce_apiserver` binary |
|
||||
| CLI tool | `nix build .#xcecli` | `xcecli` binary |
|
||||
| Admin config CLI (installable) | `/xce-build xceadmin` | `xceadmin-<version>-py3-none-any.whl` at `<build_dir>/xceadmin/dist/` — `pip install` it directly |
|
||||
| AppImage | `nix bundle --bundler github:ralismark/nix-appimage --option sandbox false .#xce` | `.AppImage` bundle (see README.md 3.5) |
|
||||
| Debug build | `nix build .#xcecore_debug` | full debug + ASAN binaries |
|
||||
|
|
@ -1,72 +0,0 @@
|
|||
# Configuration Design: What Belongs in Shared Config
|
||||
|
||||
Governs any XCE distributed-config schema decision — `ClusterConfig` (controller.capnp), and any future worker/scheduler/apiserver shared-config struct. Applies whenever a new field is proposed for a config blob that gets pushed/replicated to multiple nodes (e.g. via `OP_upgradeConfig` or an equivalent future mechanism).
|
||||
|
||||
## The Question
|
||||
|
||||
A parameter needs a home. Two choices:
|
||||
- **Shared config** — part of the cluster-wide config blob, distributed to every node, changed via a single coordinated push (build → validate → replicate → majority ACK).
|
||||
- **Local/instance config** — set per-node (CLI flag, local config file, env var), changed independently, no cluster-wide coordination.
|
||||
|
||||
Getting this wrong in either direction has a cost: putting a per-instance value in shared config forces every routine node-level change through a cluster-wide config push; putting a cluster-invariant value in local config lets nodes silently drift out of sync.
|
||||
|
||||
## Rule 0: Bootstrap availability (checked first, overrides everything else)
|
||||
|
||||
> If a parameter is needed to establish consensus, elect a coordinator, or otherwise receive the shared config in the first place, it CANNOT live inside that shared config.
|
||||
|
||||
This isn't a judgment call like rules 1–2 below — it's a structural/chicken-egg constraint. A node can't depend on config it doesn't have yet to obtain that same config. Example: an HA/Raft election heartbeat interval/timeout — a controller needs this value to participate in the election that determines who's primary and delivers the cluster config; it must be locally available (CLI flag, hardcoded default) before any config exists. This is independent of whether the field would otherwise pass or fail the independence/frequency rules below — even a field that's cluster-invariant and rarely-changing (which would normally mean "shared") is disqualified from shared config if it's needed to bootstrap access to that config.
|
||||
|
||||
If a candidate field passes Rule 0 (i.e., it isn't a bootstrap dependency), proceed to the two rules below.
|
||||
|
||||
## The Two Rules
|
||||
|
||||
Apply both to every candidate field. **Excluding on either rule is sufficient** — a field only belongs in shared config if it fails both. Note: failing both rules means shared config is the *default* home, not automatic — see "Operational-uniformity override" below for when a field that would otherwise qualify for local config gets deliberately centralized anyway.
|
||||
|
||||
### 1. Independence rule
|
||||
|
||||
> If a parameter can change on one node without requiring every other node to match, it does NOT belong in shared config.
|
||||
|
||||
Test: does divergence between nodes on this field cause a correctness problem, or is it just... fine? If fine, it's independent — keep it local.
|
||||
|
||||
### 2. Frequency rule
|
||||
|
||||
> If a parameter is expected to change more often than the rest of the config, it does NOT belong in shared config.
|
||||
|
||||
Test: would this field's natural rate of change force config pushes far more often than everything else in the blob needs? If yes, bundling it in shared config makes routine operational changes as heavyweight as a schema change.
|
||||
|
||||
### Default: both rules fail → include
|
||||
|
||||
A field that must be identical across every node AND changes rarely belongs in shared config. That's the actual reason shared config exists — to keep genuinely cluster-invariant values from drifting.
|
||||
|
||||
## Operational-uniformity override
|
||||
|
||||
Passing the independence rule (no correctness requirement to match across nodes) means a field is *eligible* for local config — it does not mean it *must* go local. A field can still be deliberately kept in shared config when centralizing it has operational value beyond pure correctness: predictable behavior across the cluster, one less per-node setting for an operator to track, avoiding "why does this node behave differently" support questions. This is a judgment call, not automatic — when exercising it, document the reasoning inline (don't just leave a field in shared config by default without stating why it's an exception).
|
||||
|
||||
Worked example: `ClusterConfig`'s `heartbeatInterval`/`inactiveHeartbeatCount` fields govern controller→**worker** liveness (when a controller marks one of its own workers unavailable) — not the HA/Raft controller-to-controller election heartbeat (see Rule 0 example above; those are two different "heartbeat" concepts sharing similar names, easy to conflate). Each controller only manages its own connected workers, so nothing breaks if this value differs node-to-node — it passes the independence rule cleanly. Decided (2026-07-20) to keep it in shared config anyway: operators want uniform, predictable worker-timeout behavior across the cluster rather than per-node drift, even though no correctness invariant demands it.
|
||||
|
||||
## Worked Examples
|
||||
|
||||
| Field | Rule 0 (bootstrap) | Independence | Frequency | Verdict | Why |
|
||||
|---|---|---|---|---|---|
|
||||
| Cluster name | passes (not a bootstrap dep) | fails — every node must agree | fails — set once at bootstrap | **Shared** | Identity mismatch across nodes is a correctness bug |
|
||||
| HA/Raft election heartbeat interval/timeout | **fails Rule 0** — needed to run the election that delivers config in the first place | n/a — disqualified before reaching this rule | n/a | **Local** (CLI flag; currently hardcoded, see roadmap) | Can't depend on config you don't have yet to obtain that config |
|
||||
| Controller→worker liveness heartbeat/timeout | passes (not a bootstrap dep) | passes — one controller's worker-timeout tolerance has no cross-node dependency | — | **Shared** (operational-uniformity override, not a rule outcome — see above) | Would qualify for local by the rules alone; kept shared for predictable cluster-wide behavior |
|
||||
| Listen port | passes | passes — each node's own bind port has no cross-node dependency | — | **Local** | Purely a per-process network setting |
|
||||
| Peer/candidate list (cluster membership) | passes | passes — adding a standby doesn't require the rest of the config to change | passes — routine operational action | **Local** (own mechanism) | Coupling membership changes to a full config push makes scaling the cluster as heavyweight as a schema migration |
|
||||
| Internal hash-table sizes, buffer sizes (perf tuning) | passes | passes — effect is local to the process that sets them | — | **Local** | No node needs to know another node's internal tuning |
|
||||
| Any monotonic version/generation counter | n/a — not operator input | n/a | n/a | **Not config at all** | Protocol state the coordinator (primary) assigns, not a value an operator sets |
|
||||
|
||||
The last row is a distinct fourth category, not a rule outcome: some fields aren't "config" in the operator-input sense — they're state the system itself maintains (e.g. a config generation counter incremented by the primary on every push). Don't force these into the shared-config-vs-local-config framing; they're neither.
|
||||
|
||||
## Applying This
|
||||
|
||||
When a new field is proposed for a shared config blob:
|
||||
1. Check Rule 0 first — if the field is needed to bootstrap/obtain the config itself, it's local, full stop, regardless of what rules 1–2 would say.
|
||||
2. Otherwise, run it through the independence rule, then the frequency rule.
|
||||
3. If it fails either, its default home is local (CLI flag, per-node config file, a dedicated dynamic-membership mechanism, etc.) — unless the operational-uniformity override applies and is explicitly documented as a deliberate exception.
|
||||
4. If it passes both (fails neither exclusion), add it to shared config — and document why in the owning spec, so the next person doesn't have to re-derive it.
|
||||
5. If it's coordinator-assigned protocol state rather than operator input, keep it out of the operator-facing config schema entirely.
|
||||
|
||||
## Origin
|
||||
|
||||
Derived 2026-07-20 while classifying `ClusterConfig`'s 21 fields for `xceadmin config build` (`.kiro/specs/admin-config-tool/`) — 6 fields were being pulled into shared YAML config by default with no principled reason to include vs. exclude them. See `brief.md`'s "Configuration Inclusion Rules" section in that spec for the full field-by-field classification of `ClusterConfig` specifically; this file holds the general rule for reuse beyond that one spec.
|
||||
|
|
@ -1,100 +0,0 @@
|
|||
# Error Handling Standards
|
||||
|
||||
## Philosophy
|
||||
|
||||
Fail fast with a consistent, machine-readable response. Handle errors at the controller boundary; let business logic return typed errors. Never silently swallow errors.
|
||||
|
||||
## Go API Response Pattern
|
||||
|
||||
All HTTP responses go through two helpers in `controllers/response.go` (a small number of legacy handlers, e.g. `FilesController` list/dir endpoints in `controllers/files.go`, still call `ctx.JSON()` directly — treat those as bugs to fix opportunistically, not as sanctioned exceptions):
|
||||
|
||||
```go
|
||||
// Success (always HTTP 200)
|
||||
ReturnSuccess(ctx, common.SUCCESS_CODE, common.SUCCESS_MSG, data)
|
||||
|
||||
// Error (HTTP status == code parameter)
|
||||
ReturnError(ctx, common.ERROR_AUTH_CODE, common.ERROR_ACCESS_FORBIDDEN_MSG, nil)
|
||||
ctx.Abort() // always Abort() after ReturnError in middleware
|
||||
return // always return immediately after ReturnError in handlers
|
||||
```
|
||||
|
||||
**The early-return pattern is non-negotiable.** Every error path ends with `ReturnError(...); return`. `ctx.Abort()` should always follow in middleware since gin does not stop the handler chain on its own — `middleware/whitelist.go`'s `WhitelistIPMiddleware` currently has one path that returns without calling `ctx.Abort()` (invalid client IP); that is a gap to close, not an accepted variant.
|
||||
|
||||
## HTTP Status Code Mapping
|
||||
|
||||
| Code | Constant | When to use |
|
||||
|------|----------|-------------|
|
||||
| 200 | `SUCCESS_CODE` | Successful operation |
|
||||
| 201 | `CREATED` | Resource created |
|
||||
| 400 | `INVALID_PARAMS_CODE` | Malformed or invalid input |
|
||||
| 401 | `ERROR_AUTH_CHECK_TOKEN_CODE` | Missing or invalid JWT |
|
||||
| 403 | `ERROR_AUTH_CODE` | Valid JWT but insufficient permission |
|
||||
| 404 | `ERROR_NOT_FOUND_CODE` | Resource not found |
|
||||
| 429 | `ERROR_TOO_MANY_REQUESTS_CODE` | Rate limit exceeded |
|
||||
| 500 | `ERROR_CODE` | Unexpected server error |
|
||||
|
||||
All code constants are defined in `common/common.go`. Add new codes there — do not inline numeric literals in handlers.
|
||||
|
||||
## Structured Logging with Zap
|
||||
|
||||
Log at the call site before returning an error. Use typed field helpers — never `fmt.Sprintf` in zap calls.
|
||||
|
||||
```go
|
||||
// Preferred pattern
|
||||
logger.Error("rate limit exceeded",
|
||||
zap.String("path", path),
|
||||
zap.String("client_ip", ctx.ClientIP()),
|
||||
zap.Int("errorcode", code),
|
||||
)
|
||||
|
||||
// For error values
|
||||
logger.Error("validate token error", zap.Error(err))
|
||||
|
||||
// For complex types
|
||||
logger.Debug("permissions resolved", zap.Any("permsWithType", permsWithType))
|
||||
```
|
||||
|
||||
**Log levels**:
|
||||
- `Error` — operation failed, client or server fault
|
||||
- `Warn` — recoverable edge case (e.g., retried operation succeeded)
|
||||
- `Info` — key lifecycle events (server start, config load)
|
||||
- `Debug` — diagnostic detail, disabled in production
|
||||
|
||||
**Never log**: JWT tokens, passwords, secret keys, or full request bodies containing sensitive data.
|
||||
|
||||
## C Error Propagation
|
||||
|
||||
Return negative errno-style codes for C daemon errors. Use the `errnox.h` definitions from `src/include/xce/`. Check and propagate immediately — do not continue on error:
|
||||
|
||||
```c
|
||||
int rc = xce_send_request(&req);
|
||||
if (rc < 0) {
|
||||
log_error("send_request failed: %d", rc);
|
||||
return rc;
|
||||
}
|
||||
```
|
||||
|
||||
**Controller daemons MUST NEVER self-terminate on an operational condition.** A controller exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`_exit()`/`abort()`/`tlog_exit()` as a designed response to an operational failure (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" here means: log a loud operator-visible ERROR (with the actionable boundary), enter a permanent non-promotable/fenced state, **stay alive, and keep retrying** so the node self-heals when conditions recover — never terminate (a self-exited controller needs manual recovery and degrades quorum meanwhile). `assert()`/abort is reserved for genuine programming-invariant violations only. Authoritative statement: `ha-async-io.md` (process-lifecycle invariant) and `src/controller/CLAUDE.md` ("Process Lifecycle"); precedent: `ha-commit-durability`'s `persist_recover_loud_fence_if_unrecoverable` + `recovery_unrecoverable` sticky fence.
|
||||
|
||||
## Input Validation Pattern
|
||||
|
||||
Validate at the controller boundary in two steps: bind the request body with `utils.BindJSONStrict`, then pass the already-bound struct pointer to a `preCheck{Resource}Request()` helper before touching services:
|
||||
|
||||
```go
|
||||
func (j JobController) CreateJob(ctx *gin.Context) {
|
||||
var req models.JobSubmitRequest
|
||||
if err := utils.BindJSONStrict(ctx, &req); err != nil {
|
||||
ReturnError(ctx, common.ERROR_CODE, err.Error(), nil)
|
||||
return
|
||||
}
|
||||
|
||||
parsedIndices, err := preCheckJobRequest(&req)
|
||||
if err != nil {
|
||||
ReturnError(ctx, common.ERROR_CODE, err.Error(), nil)
|
||||
return
|
||||
}
|
||||
// ... service call
|
||||
}
|
||||
```
|
||||
|
||||
Keep `preCheck*` functions in the same file as the handler. They take the already-bound request struct pointer (never `ctx`) and return `error`, or `(extra, error)` when the caller also needs derived data — e.g. `preCheckJobRequest(req *models.JobSubmitRequest) ([]int, error)`, `preCheckHostControlRequest(req *models.HostControlRequest) (bool, error)`. They never bind the request themselves and never return the request struct — binding is always the separate `utils.BindJSONStrict` step shown above. Note that handlers choose the response code for bind/precheck failures per-endpoint; `common.ERROR_CODE` (500) and `common.INVALID_PARAMS_CODE` (400) both appear in practice depending on the handler.
|
||||
|
|
@ -1,105 +0,0 @@
|
|||
# HA Protocol & Async I/O
|
||||
|
||||
## Philosophy
|
||||
|
||||
The HA layer combines Raft-style consensus election with a single-writer replication model. The async I/O framework (iomgr/vtaskmgr/delaymgr) enforces strict threading rules: non-blocking dispatch on the epoll thread, thread-pool handler execution, and timer-thread blocking I/O.
|
||||
|
||||
**Single-writer constraint**: only the primary writes WAL entries. This avoids Raft log replication complexity while the election mechanism inherits Raft's safety guarantees (term-based voting, freshness gates, monotonic commits).
|
||||
|
||||
**Process-lifecycle invariant (applies to ALL controller code)**: a controller MUST NEVER self-terminate. It exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`abort()`/`tlog_exit()` as a designed response to an operational condition (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" means log a loud operator-visible ERROR + enter a permanent non-promotable/fenced state + **stay alive and keep retrying catch-up** (so the node self-heals when a peer returns, without manual admin recovery) — it does NOT mean exit. See `src/controller/CLAUDE.md` → "Process Lifecycle (DESIGN INVARIANT)" for the authoritative statement.
|
||||
|
||||
## Async I/O Model
|
||||
|
||||
Three components in `src/lib/` plus a dedicated slow-lane executor:
|
||||
|
||||
| Component | Thread model | May do | Must NOT do |
|
||||
|-----------|-------------|--------|-------------|
|
||||
| **iomgr** | Single-threaded epoll loop | non-blocking socket I/O; endpoint lifecycle; **dispatch to vtask** | any blocking call; **any inline message processing** (dispatch it) |
|
||||
| **vtaskmgr** | N-thread pool (default 8, `--vcpus` CLI override; cluster.c:1743-1745) | bounded handler work; **bounded** blocking ONLY under admission reservation | **long/unbounded** blocking I/O (snapshot transfer, config-push) — starves the pool |
|
||||
| **delaymgr** | Single timer thread | fire timers; **trigger** (enqueue) work; return fast | **connect(), file I/O, any blocking** — it drives heartbeats + election timeouts |
|
||||
| **slow lane** | dedicated executor / admission-capped pool | long blocking I/O: reconnect, snapshot transfer, config-push | — (this is where blocking ops go, NOT delaymgr) |
|
||||
|
||||
**Thread-role contract (DESIGN INVARIANT — applies to ALL components: controller, worker, scheduler).**
|
||||
No thread that carries liveness signals (iomgr, delaymgr) may execute blocking work inline:
|
||||
1. **iomgr** does only non-blocking I/O + dispatch. It never blocks and never processes a message inline — it hands off to a vtask.
|
||||
2. **delaymgr** only fires timers and triggers work, then returns. It NEVER blocks (no `connect()`, no file I/O) — because the same single thread drives heartbeats and election timeouts, so any block there stalls liveness for the whole cluster and can cause spurious failover.
|
||||
3. **vtask pool** runs bounded handler work. **Bounded** blocking (e.g. the commit-wait barrier) is permitted ONLY under an admission reservation (`COMMIT_WAIT_RESERVED`, `wait_cap`/`wait_admit_locked`, commit.c:241-271) that keeps workers free for liveness-critical messages. **Long/unbounded** blocking I/O does NOT belong here.
|
||||
4. **Long blocking I/O** (reconnect, snapshot transfer, config-push) runs on a dedicated slow lane / admission-capped executor, never on iomgr or delaymgr.
|
||||
|
||||
> **SUPERSEDES (2026-06-25):** the former rule "delaymgr: connect()/file I/O = Permitted; blocking calls belong in delaymgr tasks." That guidance was unsafe — delaymgr is *also* the heartbeat/election-timeout driver (see Heartbeat below), so a permitted blocking `connect()` there stalls heartbeats to healthy peers → spurious election. Blocking ops move to the slow lane; delaymgr becomes trigger-only. A blocking audit (2026-06-25) found three live violations of the corrected contract — snapshot send + `persist_checkpoint` inline on iomgr (`ha.c:2702`), `push_config_transport_only` busy-poll+join on iomgr (`config_repl.c:1332`), and blocking `gethostbyname`+`connect` on the delaymgr thread (`netio.c:289/302`). **All three are now fixed** — spec `raft-drive-async` (IMPLEMENTED, validate-impl GO 2026-07-18, 21/21 tasks) moved each off its liveness thread: snapshot transfer now runs on a dedicated slow-lane worker (`snapshot_lane_fn`, `ha.c:4021`, kept off the iomgr thread — see the "DEFAULT ON" note at `ha.c:5568`); `push_config_transport_only` now spawns a dedicated pthread per standby (`pthread_create`, `config_repl.c:1589`) instead of busy-polling+joining on iomgr; and `socket_connect` (`netio.c:302`, doc comment `netio.c:287`) is now a bounded, non-blocking TCP connect via `poll()` with `connect_timeout_ms`, so it no longer pins the delaymgr heartbeat thread. See bug-1447 below for a same-day crash this spec's own validation soak surfaced and fixed.
|
||||
|
||||
Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — always non-blocking. Replies use `xce_reply(ep, host, &reply)` — also non-blocking.
|
||||
|
||||
**Same-batch endpoint lifetime (bug-1447, fixed 2026-07-18).** `io_worker`'s epoll loop processes every fd from ONE `epoll_wait()` batch (`events[MAX_EPOLL_EVENTS]`) in a single pass. A handler for one event in that batch (e.g. `on_standby_connected` evicting a stale duplicate outbound channel via `iomgr_unmonitor()`+`endpoint_unref()`) can free a DIFFERENT endpoint that ALSO has a ready event later in the SAME batch array — `epoll_ctl(DEL)` only stops *future* `epoll_wait()` calls, it does not invalidate an event already sitting in the current batch. The later event then dereferences a freed `endpoint_t`/`SSL*` (heap-use-after-free). `INVOKE_MSG_HANDLER`'s ref/unref bracket only protects the endpoint a handler is CURRENTLY running on, not a different endpoint torn down as a side effect. Fix: `io_worker_dispatch_batch()` takes `endpoint_ref()` on every endpoint in the batch before dispatching any of them, and `endpoint_unref()`s each only after the whole batch completes — same lock-free bracket idiom as bug-1298, applied to the batch as a unit. When adding any handler that can tear down an endpoint other than its own, this batch-wide ref bracket is why that's safe — do not "optimize" it away as redundant with `INVOKE_MSG_HANDLER`'s per-call ref.
|
||||
|
||||
## Connection Ownership
|
||||
|
||||
**Inbound vote connections** (accepted on the listener): Used **exclusively** for vote RPC exchange (RequestVote + reply). After the reply is sent, the connection is left open but NOT reused for state push. Owned by the remote peer (caller closes).
|
||||
|
||||
**Outbound control channels** (`ha->standby_endpoints`): Carry all persistent state from primary to standby — ctrlhello, config push, WAL replication, heartbeat NOPs. Owned by `vote_ep_err_handler` (promoted vote endpoints) and `standby_ep_err_handler_primary` (late-joiner outbound connections). Freed on disconnect.
|
||||
|
||||
**Late joiners**: `primary_heartbeat_fn` scans the peers list and opens a fresh outbound connection to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello, registers the endpoint, and calls `repl_add_standby`.
|
||||
|
||||
## Raft Election
|
||||
|
||||
The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed verbatim** (election, `logOk` freshness, `(term,index)` truncation, current-term commit, `UpdateTerm`). The controller (`ha.c`/`cluster.c`) is a **thin adapter** over it via five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`). The controller does NOT re-implement Raft protocol decisions — landed by spec `ha-term-safety` (closes bug-851: silent committed-prefix divergence from term-blind `hdr.term==0` logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path.
|
||||
|
||||
**Vote-request send failure MUST still reach the reply tally (bug-1423/1424).** `raft_cb_send_vote_request` / `raft_cb_send_pre_vote_request` send RequestVote/PreVote to every OTHER configured peer (dead or alive) and feed replies back to the library via `vote_reply_dispatch`/`pre_vote_reply_dispatch` (→ `raft_on_vote_reply`/`raft_on_pre_vote_reply`). A `endpoint_connect`/`iomgr_monitor` failure (the normal outcome when a peer is down) is itself an outcome that must be dispatched as a "not granted" reply through that SAME production path — never silently dropped, and never routed through the legacy test-only `handle_request_vote_reply`. Without this, the library's `replies_received == n_peers - 1` fast path ("everyone answered, decide now") is structurally unreachable whenever any peer is down, and recovery falls back entirely on the raw per-campaign timeout — real-election observed 2+ minutes permanently stuck in `CTRL_ELECTING` in a 3-node cluster with one node down before this fix.
|
||||
|
||||
- **RequestVote + RequestVoteReply** over Cap'n Proto
|
||||
- **Freshness gate** (`(term,lsn)`, term-primary): a voter grants iff `cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len)`. The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below).
|
||||
- **Votes persisted atomically** with term increment (raft_cb_persist)
|
||||
- **Majority wins**: quorum = ceil(n_peers / 2)
|
||||
- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised `(tag, durable-floor LSN)` MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to `self_lsn` and truncates committed entries — bug-828).
|
||||
- **WAL truncation rule (load-bearing — bug-828)**: every `persist_truncate_wal_above` call site must cut at a value that is provably safe — never at the lagging learned-committed watermark (`last_known_committed_lsn`), since it lags the true commit point and a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). Four production call sites, all audited safe (2026-07-09): (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile, both running *before* the new leader commits anything; (c) `ha_demotion_convergence_fn` (`ha_convergence.c`), which truncates to the node's own local `committed_lsn` on demotion-triggered restart — a safe, never-stale local watermark, not the cross-node ctrlhello field this rule warns about; (d) `persist_wal_write_replicated`'s same-LSN payload-conflict repair (`persist.c`, `truncate_wal_above_locked(p, lsn-1)`), which removes exactly one divergent tip entry before re-appending the authoritative one at the same LSN and restores `committed_lsn` verbatim afterward — never regresses the committed boundary. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello".
|
||||
|
||||
**Key safety invariants**:
|
||||
- `currentTerm` is monotonic (never decreases)
|
||||
- `votedFor` updated atomically with term increment before granting any vote
|
||||
- `commitIndex` never decreases (monotonic)
|
||||
|
||||
## Heartbeat Model
|
||||
|
||||
Bidirectional heartbeat with Raft-style AppendEntries semantics:
|
||||
|
||||
- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
|
||||
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag, term)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
|
||||
- **Heartbeat-reply term-discovery fallback (bug-1421).** A stale-term primary does NOT reliably learn a higher term from an AE reject: `ha_raft_on_repl_ack` only routes AE replies through `raft_on_append_entries_reply` on `success=1`, and that function itself drops any reply whose term doesn't match (`raft.c` stale-term drop) — so an AE reject can never adopt a higher term through the normal path. `stateReport.term` (above) and a WAL-NACK's carried term are the two fallback vectors: both call `raft_on_higher_term_observed(s, observed_term)` (strictly-greater-than only; touches `term`/`voted_for`/`role`; `on_persist` before `on_become_follower`), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever.
|
||||
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`. **Every return path must re-arm (bug-1562).** The 5.10b RDHUP-cooldown debounce (suppresses a fresh failover within `2 * election_timeout_ms` of the last trigger — a dead primary's fd RDHUPs faster than an election settles) checks only elapsed time, not whether an election is genuinely still in flight, unlike the sibling 5.6 guard (`role == CTRL_ELECTING || pre_vote_in_progress`). A debounced RDHUP landing after the in-flight round already concluded — e.g. a lone cold-booting node whose first PreVote loses in milliseconds with no peers up yet — must still call `ha_arm_heartbeat_timeout_vtask` before returning, or the watchdog is never rescheduled and the node freezes in CTRL_STANDBY permanently (live-reproduced: a 3-node cluster with a long inter-node start stagger never elected a primary).
|
||||
|
||||
## Replication Consistency (Two-Tier)
|
||||
|
||||
**Tier 1 — Majority (service gate)**: Every state-changing operation is committed on a majority before acknowledgment. Post-election write safety comes from Raft's own-term commit rule (§5.4.2): an election winner assumes `CTRL_PRIMARY` directly and, at promotion, appends a `WAL_NOOP` entry of its current term and arms the commit engine's writable-LSN gate (`commit_set_writable_lsn`). Confirmation of any operation then waits behind `committed_lsn >= writable_lsn` — no wall-clock path changes leadership or confirms operations. Single-node mode returns -ENODEV (degraded).
|
||||
|
||||
**Tier 2 — Full peers (continuous best-effort)**: Primary continuously pushes config and WAL to **all** standbys via `ha_dispatch_repl_action_for_standby` on each heartbeat reply. The dispatch gate in CTRL_INIT and CTRL_ELECTING allows replication opcodes so campaigning nodes never reject state push.
|
||||
|
||||
## Per-Entry Term & `(term,lsn)` Reconcile (ha-term-safety)
|
||||
|
||||
Every WAL entry carries the **leader's real Raft term** in `wal_entry_hdr_t.term` (a value fix — the 40-byte header layout is unchanged). The primary stamps `s->term` via `raft_log_append` → `on_persist_entry`; the **live replication carries that term** (the legacy term-0 `repl_broadcast` was retired — a standby persists the *leader's* term, not 0 or a local substitute). Standbys intake every heartbeat/replication through `raft_on_append_entries_request`, so `UpdateTerm` (adopt a higher term + step down) and `(term,lsn)` conflict-truncate happen on every RPC. **Reconcile by `(term, lsn)`, never by LSN alone** — a same-LSN entry with a different term is divergent and is truncated, then re-accepted from the leader. Recovery reads back `hdr.term` per LSN (legacy pre-feature term-0 maps to a single "lowest-known" term for matching). Configuration changes ride the **same unified log** as a `WAL_CONFIG_MARKER` (a `{version_tag, sha256}` reference, no config bytes) that commits + reconciles by `(term,lsn)` exactly like a job entry; the config *bytes* travel out-of-band and are **hash-verified against the committed marker before activation** (mismatch ⇒ loud serve-fence, stay alive + retry).
|
||||
|
||||
## Commit-Source Ownership (a deliberate decision — do not re-add a library commit ceiling)
|
||||
|
||||
The **durable-majority barrier + the own-term `WAL_NOOP` writable gate** (`commit.c`, see `persistence.md`) is **authoritative** for advancing `committed_lsn`. The raft library's `commit_index` (fed via `on_commit_advanced` after `raft_advance_commit`) is kept **advisory only** — it is NOT folded in as a hard ceiling (`committed = min(durable-majority, raft_commit_index)` was tried and removed). Rationale: under the single-writer model the durable-majority watermark is *already* a current-term `(term,lsn)`-safe quorum (term-carrying replication + reconcile-before-durable + the own-term gate), so the library index is redundant for safety; and because the library index advances asynchronously and lags whenever a live path (e.g. catch-up delivery) under-drives it, a `min()` ceiling **starves commit below the safe durable majority** (jobs acked-durable on a majority but never committed). Keep driving the library (for its own `match_index`/`next_index`/send state) but never let its commit index *gate* `committed_lsn`. The Figure-8 "current-term only" guarantee is realized by single-writer + the own-term NOOP gate (a new leader cannot ack a prior-term entry as committed until its own-term NOOP is majority-durable), not by the library ceiling.
|
||||
|
||||
## Dispatch Gate
|
||||
|
||||
`ha_dispatch_gate` controls which opcodes are accepted per role:
|
||||
|
||||
| Role | Allowed |
|
||||
|------|---------|
|
||||
| **CTRL_PRIMARY** | All opcodes, subject to two fences checked before the quorum gate (`ha_dispatch_gate`, ha.c:4525): `config_serve_fenced()` (ha-term-safety task 4.3) and `upgrade_materialization_fenced()` (config-upgrade-snapshot task 6.2) each refuse state-changing ops while set, admitting only replication/read-only/status opcodes. Once unfenced: client ops are subject to write quorum; under no quorum, the fact allowlist `OPCode_notify/update/setpreason/clrpreason` (plus read-only/status/replication opcodes) is still admitted |
|
||||
| **CTRL_STANDBY** | nop, reply, hello, join, shutdown, heartbeat, redirect, steppingDown, requestVote, preVoteRequest, probeConfigTag, upgradeConfigAck, ctrlstatus, + all replication opcodes (replEntry, replAck, replSnapshot, replCatchUpDone, replConfig, replConfigPush, walAck, snapshotTransfer, walCatchupEntry) |
|
||||
| **CTRL_INIT / ELECTING** | requestVote, preVoteRequest, probeConfigTag, heartbeat, hello, nop, replConfigPush, walCatchupEntry, snapshotTransfer, ctrlstatus |
|
||||
|
||||
This ensures a campaigning node never rejects state push from the current primary — rejecting during an election window would leave that node permanently behind.
|
||||
|
||||
## Raft-to-Code Mapping
|
||||
|
||||
| TLA+ Variable | XCE Code |
|
||||
|--------------|----------|
|
||||
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) |
|
||||
| `state[i]` | `ha->role` (INIT, ELECTING, PRIMARY, STANDBY) |
|
||||
| `votedFor[i]` | `ha->voted_for_name[]` |
|
||||
| `log[i]` | WAL segments in `persist->wal_path` |
|
||||
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) |
|
||||
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) |
|
||||
|
|
@ -1,96 +0,0 @@
|
|||
# WAL & Persistence Durability
|
||||
|
||||
How the controller turns appended state into durable, recoverable, majority-backed state. This is the persistence-layer companion to `ha-async-io.md` (HA protocol / replication) and the `src/controller/CLAUDE.md` Protocol Verification Checklist (the authoritative safety gate). Owning code: `src/controller/persist.c`, `src/include/persist.h`, `src/controller/commit.c`. Landed by spec `ha-commit-durability` (2026-06-18, closes bug-661).
|
||||
|
||||
## Core Invariant: Acknowledged ⇒ Durable on a Majority
|
||||
|
||||
When the controller replies *ok* to a client, the operation's WAL entry is `fdatasync`'d on a fault-tolerant majority (the acking member + ⌊N/2⌋ others). The reply path is unchanged — it is gated on the commit barrier; the durability guarantee comes from strengthening what "committed" *means*, not from a new gate.
|
||||
|
||||
The load-bearing rule: **a member contributes to the commit majority for an LSN only once that LSN is durable locally.** The primary counts its own copy at `durable_lsn`, never at the appended tip.
|
||||
|
||||
## `durable_lsn` — The Prefix Watermark
|
||||
|
||||
`persist_t.durable_lsn` (`_Atomic uint64_t`) is the highest LSN whose bytes are `fdatasync`'d locally. It is a **prefix watermark**, never a per-entry set: `durable_lsn ≥ L` implies every entry `≤ L` is durable (because the WAL is a contiguous single-writer log). Invariants:
|
||||
|
||||
- `durable_lsn <= current_lsn - 1` (you cannot be durable past what you have appended).
|
||||
- Monotonic non-decreasing **except** clamp-on-truncate (`persist_truncate_wal_above` clamps `durable_lsn` down to the new tip). **Because the clamp lowers `durable_lsn`, a truncate cutoff that lags the true commit point physically destroys committed entries** — only ever call `persist_truncate_wal_above` with the votes-derived post-election `safe_committed_lsn` or the library `(term,lsn)` conflict point, NEVER with the lagging `last_known_committed_lsn` (bug-828).
|
||||
- Advances **only after a successful `fdatasync`** — fail-loud: on `fdatasync` error the watermark holds and an ERROR is logged. Never advance a watermark on a failed sync.
|
||||
|
||||
Read it via `persist_durable_lsn(p)`. The commit engine's self-cap (`majority_candidate_locked` in `commit.c`) uses exactly this value.
|
||||
|
||||
## Group-Commit Flusher (the `fsync_worker` pattern)
|
||||
|
||||
`fsync_worker` is **demand-driven, not a timer.** On append, `persist_wal_async` pwrites under `p->lock`, releases, then signals the flusher. The flusher does **one** `fdatasync` covering all pending appends, advances `durable_lsn`, then wakes commit evaluation. This amortizes one sync across the high-volume `WAL_MODE_RETAIN` fact stream (job start/finish/fail/rusage).
|
||||
|
||||
Two correctness rules when touching this path:
|
||||
|
||||
- **Lost-wakeup guard is level-triggered** on the appended-vs-synced gap (`durable_lsn < current_lsn - 1`), re-checked under `p->lock` before waiting and after each flush — never edge-triggered. An entry must never be considered durable before its `fdatasync`.
|
||||
- **A sustained `fdatasync` failure must still back off (bug-1563).** The level-triggered gap above can't distinguish "fresh work just landed" from "still stuck from the last failed attempt" — both read as pending. `fsync_worker` skips its wait ONLY when the just-completed flush made genuine progress (succeeded) or there was nothing pending to begin with; a flush that WAS pending but made no progress (failed) still waits out the normal `wal_sync_ms` cadence before retrying. Without this, a sustained failure (real disk error, or this sandbox's overlayfs quirk) spins `flusher_flush_once`/`fdatasync` as fast as the CPU/lock allow — a live 40-round chaotic soak hit 300k+ failed attempts on one node in under a minute (one core pinned, log flooded) before this fix.
|
||||
- **Commit wake-up via callback, not a back-reference.** Both the flusher AND `persist_flush` (below) call `on_durable_advance(ctx)` (registered with `persist_set_durable_cb`, ctx = the `commit_engine_t`) **only when `durable_lsn` advanced, and after releasing `p->lock`**. Lock order is strictly `p->lock → ce->lock` (cycle-free; persistence never includes commit). On shutdown, **join the flusher before** `commit_engine_fini` or the callback can fire into freed state. **`persist_flush` firing this callback is itself a fix (bug-1564)** — it used to advance `durable_lsn` silently, so a WAIT-blocked commit caller was only ever woken by the async flusher's own next successful cycle, never by a synchronous flush. The gap was latent until bug-1563's backoff fix above removed an accidental cover: pre-1563, the flusher's un-throttled retry loop happened to notice any state change and re-fire the callback itself within microseconds, masking that `persist_flush`'s own advance was unobserved.
|
||||
|
||||
`--wal-sync-ms` (default `DEFAULT_WAL_SYNC_MS = 10`) is the **maximum batching delay**, a latency/throughput tunable — it is *never* a durability gap. An acked op is durable regardless of its value (a large value just widens the batch window; the commit barrier still waits for `durable_lsn`).
|
||||
|
||||
## Standby Persist-Before-ACK
|
||||
|
||||
A standby `fdatasync`s a replicated entry before it sends its WalAck. `persist_wal_write_replicated` rc-gates the inline sync and returns `-EIO` on failure; the caller only ACKs on `rc == 0`. A standby must never ACK a non-durable entry — that is half of the majority guarantee.
|
||||
|
||||
## Recovery Integrity Verdict
|
||||
|
||||
`replay_wal` / `persist_recover` distinguish a clean log end from corruption and surface a `recover_verdict_t`:
|
||||
|
||||
- `RECOVER_CLEAN` — valid CRC chain to a clean entry boundary, OR an empty first-boot WAL, OR a pre-allocated-zero tail, OR a snapshot sentinel + clean replay.
|
||||
- `RECOVER_TRUNCATED` — replay stopped on structural corruption (CRC chain break, CRC mismatch, short header/payload read).
|
||||
|
||||
The verdict is **monotonic** (init CLEAN; only ever set TRUNCATED; a later clean segment cannot un-set it) and is **structural integrity only** — "behind the cluster" is a separate concern handled by the freshness/promotion fence. Recovery never silently discards committed entries. Consume via `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` (these are NOT `XCE_TESTING`-gated — the promotion gate reads them in production).
|
||||
|
||||
## Durable Floor & Durable-Backed Freshness
|
||||
|
||||
On recovery, the member publishes its **durable floor** = highest intact replayed LSN (the WAL tip) via `persist_durable_floor(p)`. It is **assigned, not max'd**, so a truncating restart lands at the lower intact tip and never over-advertises. This floor is a durable *last-log index*, **not** the commit position.
|
||||
|
||||
Election freshness is then durable-backed (formula lives in `ha.c`/`cluster.c`, consumed from here):
|
||||
|
||||
- A **candidate advertises bare `persist_durable_floor`** for the LSN axis — it may only claim a position it can durably back.
|
||||
- A **voter floor = `max(durable_floor, last_known_committed_lsn)`** — durable tip with the committed-high-water backstop retained (bug-339).
|
||||
|
||||
Result: a truncated/behind node cannot win an election or pass the promotion gate; the most-durable reachable node always can (no livelock).
|
||||
|
||||
## What Persistence Does NOT Persist
|
||||
|
||||
- **Never write `committed_lsn` to disk.** Raft keeps the commit index volatile; durability is about the **log**, not the commit index. The post-election safe-commit point is re-derived from voter replies (`ha_compute_safe_committed_lsn`), never from a stale ctrlhello.
|
||||
- **LSN 0 is a reserved sentinel** — never a real entry; the first append is LSN 1. The commit engine rejects `lsn == 0`.
|
||||
|
||||
## Apply Correctness: `APPLY_DIVERGENT` (materialized state vs. WAL, not durability)
|
||||
|
||||
A **separate** verdict from `recover_verdict_t` (above): `persist_apply_verdict(p)` (`APPLY_CLEAN`/`APPLY_DIVERGENT`) answers "does the live materialized state (job/queue tables) actually match what a clean replay of the durable WAL would produce" — durability says the *bytes* are safe; this says the *applied result* is trustworthy. Computed once, at boot, in `persist_recover()`: an entity-count check (WAL_JOB_SUBMIT entries scanned vs. live table growth) plus a position check against `p->last_applied_lsn` (the LOGICAL apply cursor — never the physical WAL-scan tip, which also advances on checkpoint-marker/unknown-type entries and so falsely triggers DIVERGENT on an ordinary crash-during-checkpoint, bug-1419).
|
||||
|
||||
**Unconditional, restart-only fence (deliberate — landed by `raft-apply-decoupling` task 5.1-5.3, closes bug-1418's promotion-gate liveness-deadlock family):** once `APPLY_DIVERGENT`, the promotion gate (`ha.c` `raft_cb_become_leader` and `promotion_fence_timeout_fn`) refuses `CTRL_PRIMARY` **unconditionally** — no peer/HWM/durable-floor condition may clear it live, since a divergent local materialization can't be trusted for a peer/catch-up comparison either. Only a fresh, clean `persist_recover()` replay clears it (`apply_verdict` has exactly two production write sites: `persist_init()`'s reset and `persist_recover()`'s boot-time computation — no live path resets it). **Do not "fix" an apply-divergent liveness stall by relaxing this gate** — the actual fix is eliminating the false-positive DIVERGENT verdicts at the source (a boot-time TOCTOU where a returning node's verify pass raced the live apply pump / live-replication apply path, bug-1416, closed via `persist_t.recovery_complete` + `persist_mark_recovery_pending` called before any network traffic is processed).
|
||||
|
||||
**`APPLY_CLEAN` is entity-count + cursor-position only — it does NOT prove per-field completeness (bug-1678).** `persist_apply_verdict()`'s two checks (WAL_JOB_SUBMIT-count-vs-table-growth, `last_applied_lsn` position) both passed while a job's `allocs` (the resource allocation the scheduler assigned) silently replayed as `NULL` after a full restart — `job.state` was correctly `RUN`, the row existed, the cursor was in the right place, and `APPLY_CLEAN` reported healthy the entire time. Root cause: `WAL_JOB_DISPATCH`'s payload (`wal_jobstate_t`) never carried the allocation at all, so no apply-side check could have caught the gap even in principle — the field was never in the log to compare against. **The real invariant `APPLY_CLEAN` should imply — but today only partially enforces — is: online-apply and replay-apply of the same committed log MUST produce byte-identical materialized state for every persisted field of every entity, not just matching row counts and an advancing cursor.** When adding a new WAL fact type or a new field to an existing one, verify explicitly that (a) the field is actually encoded into the WAL payload (not just the in-memory struct at dispatch time), and (b) both apply seams — `apply_committed_entry()` (primary boot replay) and `apply_wal_entry()` (standby catch-up / replicated-apply) — decode and adopt it. Tracked as unimplemented future work at the systemic level in `roadmap.md`'s raft-apply-decoupling item #4 / raft-sim-conformance follow-up (a generic per-field snapshot-diff regression harness); until that lands, treat this as a standing audit dimension (see `xce-audit-invariants` skill) rather than a one-time fix.
|
||||
|
||||
## Snapshot Save/Load Field-Completeness (Scheduler)
|
||||
|
||||
**The identical invariant applies to the scheduler's own, separate persistence mechanism** (`xce_sched_core_save_snapshot()`/`xce_sched_core_load_snapshot()`, `src/scheduler/src/snapshot{,_wire,_load}.c`) -- a completely different code path from this file's WAL-based durability (no shared code, no shared fixes), but the same failure shape: a snapshot-restored entity can report the *correct state* while silently missing a *field* that state depends on, with nothing in the load path able to detect it because the field was never written to the wire type in the first place. Confirmed instance: `job->allocs`/`job->accounts` (a running job's resource allocation) were never captured by `build_wire_job()` -- `snapshot_wire.c:167`'s own comment flags it -- so a snapshot-restored RUNNING job showed the right `state` but permanently zero resource usage, the scheduler-side mirror of this section's `job->allocs`/EXEC_HOST story above (bug-1678). Scoped as its own spec, `scheduler-snapshot-job-allocation`, rather than folded into WAL work, since the two persistence mechanisms don't share fixes. Unlike the WAL side, `snapshot.c`'s own "Scope decisions" header already documents SOME gaps explicitly (`job->priority`, queue `project_name`/`job_pick_plugin`) -- those are fine, the invariant is about UNDOCUMENTED gaps. Tracked as `xce-audit-invariants`' scheduler dimension 4 -- when adding a new field to a snapshot-round-tripped entity, verify explicitly that it's written in the matching `build_wire_*()` function AND read back in `xce_sched_core_load_snapshot()`, or add it to the documented-gaps list if omitting it is a deliberate choice.
|
||||
|
||||
## Fail-Loud, Never Self-Exit
|
||||
|
||||
When a member cannot reach a committed-consistent state (recovery-truncated/behind, no healthy peer), it enters a **loud permanent fence**: log a loud operator-visible ERROR (truncation boundary vs cluster HWM), set the sticky `recovery_unrecoverable` flag, hold non-promotable, **stay alive, and keep retrying** — it self-heals when a peer returns. It MUST NEVER `exit`/`abort` as a designed response. The decision is the pure `persist_recover_decision` → `RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}`; the action is `persist_recover_loud_fence_if_unrecoverable` (no terminal). Authoritative statement: `error-handling.md` + `src/controller/CLAUDE.md` "Process Lifecycle".
|
||||
|
||||
## Testing Durability Deterministically
|
||||
|
||||
- **Force durability synchronously** with `persist_flush(p)` (does an `fdatasync`, advances `durable_lsn`, and fires `on_durable_advance` on genuine advance — bug-1564) when a test needs `durable_lsn ≥ L` before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming `durable_lsn` to the appended tip). A WAIT-blocked commit caller now wakes from either path — don't assume only the async flusher can release one.
|
||||
- **`XCE_TESTING` seams** (e.g. `persist_test_set_force_fsync_fail`, `persist_test_durable_lsn_on_disk`) are compiled out of production — verify exclusion with `nm`.
|
||||
- **Gate test runs on the ANSI-stripped `:FAIL:` count** from the test binary's stdout, not on meson's `Fail:0` summary (meson aggregates and hides Unity sub-group failures).
|
||||
- Run final durability validation on a **real fsync-able filesystem** — some sandbox `/tmp` mounts mis-handle `fdatasync`.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Concern | Symbol / Site |
|
||||
|---------|---------------|
|
||||
| Durable watermark | `persist_durable_lsn(p)` / `persist_t.durable_lsn` |
|
||||
| Durable floor (WAL tip) | `persist_durable_floor(p)` |
|
||||
| Commit self-cap | `majority_candidate_locked` (commit.c) caps self at `durable_lsn` |
|
||||
| Group-commit wake (async + sync) | `persist_set_durable_cb` → `on_durable_advance(ctx)`, fired by both `fsync_worker` and `persist_flush` on genuine advance |
|
||||
| Synchronous flush (tests) | `persist_flush(p)` — also fires `on_durable_advance` (bug-1564) |
|
||||
| Recovery verdict | `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` |
|
||||
| Loud fence (no exit) | `persist_recover_loud_fence_if_unrecoverable`, `recovery_unrecoverable` |
|
||||
| Flush cadence (max batch delay) | `--wal-sync-ms`, `DEFAULT_WAL_SYNC_MS = 10` |
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
# Product Overview
|
||||
|
||||
XCE is an HPC (High-Performance Computing) cluster management system providing job scheduling, queue management, and cluster resource control. It targets organizations running batch workloads on on-premises or air-gapped compute clusters who need deterministic scheduling, multi-tenant access control, and zero external database dependencies.
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
1. **Job lifecycle management** — submit, start, stop, restart, kill, and monitor jobs including job arrays. Resource usage data is collected and exposed via API.
|
||||
2. **Queue and scheduling** — plugin-based scheduling engine for job-to-node placement with queue control (enable/disable) and expression-based resource filtering.
|
||||
3. **Cluster and host management** — node status monitoring (OK, UNAVAIL, LOCKEDU, LOCKEDW, BUSY, CLOSED, CLOSED_BY_ADMIN), cluster topology tracking, and host control operations.
|
||||
4. **Multi-tenant RBAC + pluggable auth** — two-part `action:operation` permission IDs (e.g., `job:view`) with a separately-keyed domain scope per role (e.g., `type: self`, `type: self_ug`, `type: all`). Auth backends: configfile, LDAP, OAuth2, PAM. Built-in IP whitelist and rate limiting.
|
||||
5. **Controller persistence and HA** — WAL + checkpoint-based single-controller persistence with Raft-style quorum election (RequestVote, term-based voting, freshness gates) for standby failover and live replication over TCP (Cap'n Proto serialization).
|
||||
|
||||
## Target Use Cases
|
||||
|
||||
- **Local HPC batch computing** — long-running scientific/engineering workloads requiring deterministic scheduling, queue priority, and resource isolation.
|
||||
- **Air-gapped / secure environments** — standalone deployments with file-based auth + SSL, no external cloud dependencies.
|
||||
- **Multi-team shared clusters** — multiple teams sharing a cluster through RBAC with JWT token authentication.
|
||||
- **No-shared-storage HPC** — controller uses local filesystem (ext4/xfs) for storage; job files transferred via API upload/download rather than network shares.
|
||||
|
||||
## Value Proposition
|
||||
|
||||
XCE provides a **fully integrated single-vendor stack** (no database setup, unified build system) with an open, buildable codebase (Nix flakes, AppImage packaging). Compared to Slurm/PBS, it eliminates the external database requirement. Compared to commercial LSF, it is open source and reproducibly buildable.
|
||||
|
||||
**Key architectural trade-off**: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via the `src/lib/raft/` shared module (`raft.c`/`raft.h`, consumed by the controller through `ha_init_raft`), and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, `committed_lsn` ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
|
||||
|
||||
## Active Development Context
|
||||
|
||||
The project is in a **persistence and HA transformation** (branch `persistent_ha`). Phases:
|
||||
|
||||
| Phase | Goal | Status |
|
||||
|-------|------|--------|
|
||||
| P0 | Restart survival (WAL + checkpoint) | Complete |
|
||||
| P1 | Standby election and takeover (Raft-style) | Complete |
|
||||
| P2 | Live replication primary → standby (config + WAL) | Complete |
|
||||
| P2.5 | Worker/scheduler auto-reconnect on failover | In progress |
|
||||
| P3–P4 | Hardening, metrics, docs | In progress |
|
||||
|
||||
## Spec-Driven Development
|
||||
|
||||
Features are designed and implemented via the kiro spec system (`.kiro/specs/`).
|
||||
Each feature follows a 3-phase approval workflow: Requirements → Design → Tasks → Implementation.
|
||||
|
||||
**Status as of 2026-07-18**: the ha-consistency-invariants umbrella (ha-commit-engine,
|
||||
ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) is
|
||||
COMPLETE (closed 2026-07-13). The Raft TLA+ Conformance family (raft-library,
|
||||
raft-log-replication, raft-commit-safety, raft-sim-conformance) and its
|
||||
XCE-integration umbrella ha-term-safety are implemented + validate-impl GO.
|
||||
job-array-single-wal-record shipped complete (2026-07-17, including a
|
||||
user-directed residual-gap closure pass). A backlog sweep (2026-07-18) ran
|
||||
`/kiro-validate-impl` on 9 specs whose tasks were 100% done but had never
|
||||
been formally closed — raft-drive-async, ha-wal-replication,
|
||||
ha-heartbeat-committed-propagation, ha-prevote, ha-elect-ctx-lifecycle,
|
||||
ha-entity-snapshots, ha-commit-durability, libxce-failover,
|
||||
raft-sim-conformance — all now GO. That sweep surfaced one real bug
|
||||
(bug-1447: same-batch endpoint use-after-free in `io_worker`'s epoll
|
||||
dispatch loop, see `ha-async-io.md`), root-caused and fixed the same day.
|
||||
|
||||
Remaining backlog (requirements-only, design-pending, or partially implemented):
|
||||
admin-config-tool (spec finalized), config-upgrade-snapshot (tasks generated,
|
||||
unblocked, implementation nearly complete). scheduler-framework was decomposed
|
||||
into scheduler-core (implementation-complete), scheduler-allocation, and
|
||||
scheduler-preemption (both active). See `roadmap.md` for dependency order
|
||||
and status.
|
||||
|
||||
Key design decisions already locked:
|
||||
- No `fork()` for snapshots (mutex inheritance deadlock in multithreaded controller)
|
||||
- Raft + CP adopted (2026-06-10, supersedes earlier "No Raft"): Raft is the consensus model for both election and log commitment — majority-ACK commit barrier on all state-changing writes, reject (never degrade) when quorum is unavailable. Single-writer log retained as XCE's simplification: the controller is the sole log writer, avoiding concurrent-proposal complexity while inheriting Raft's safety rules (see `.kiro/specs/ha-consistency-invariants/brief.md`)
|
||||
- No BoltDB (using dual zarray COW for application-level checkpointing)
|
||||
- No application-layer encryption (trust OS boundary: chmod 0700, LUKS/dm-crypt)
|
||||
- No WAL logging for job deletion (deferred to checkpoint time to simplify replay)
|
||||
|
|
@ -1,122 +0,0 @@
|
|||
# Roadmap
|
||||
|
||||
## Overview
|
||||
|
||||
XCE is in the persistence + HA transformation (branch `persistent_ha`). Governing model: Raft consensus + CP (see `product.md` key decisions). Active spec work is tracked per-spec in `.kiro/specs/`; this file is the **single consolidated future-work document** — the forward-looking queue, long-deferred items, and hardening backlog. It absorbs the former `src/docs/FUTURE_WORK.md` (2026-05-12, config-replication-design review), which now redirects here.
|
||||
|
||||
## In Flight
|
||||
|
||||
- ~~**scheduler-framework** — NEW 2026-07-06 (`/kiro-discovery`). Iterator-driven allocation architecture: lazy plan iterator + host iterator, combinatorial parallel job allocation, configurable job picker, session-level unsatisfiable resreq cache. Brief: `.kiro/specs/scheduler-framework/brief.md`.~~ **DECOMPOSED 2026-07-27** into three child specs (`.kiro/specs/scheduler-framework/spec.json` `child_specs`): **scheduler-core** — core/driver separation, pluggable job picker interface, and simulation mode; **IMPLEMENTATION-COMPLETE**, GO/NO-GO validation 2026-08-07/08 (2 non-blocking follow-ups: (a) tlog logging-integration built but never activated, (b) `fcfs.c`/`builtin_fcfs.c` doc+traceability gap — design.md text was corrected same-day, but tasks 2.1-2.6 in `tasks.md` still carry no `_Verified:` lines, **still open**), 45/45 sub-tasks done as of that validation. **Updated 2026-08-19**: task 12.5 (REPL live-operator-testing fixes, Req 5.22 amendment) was added after that validation from live-operator testing gaps (bug-1593/1594/1595/1596) and is now also done, independently re-verified with fresh evidence (`/kiro-verify-completion`) — see `tasks.md`'s own `_Verified:` line on that task; this closes follow-up (a) too (the tlog gap is now activated — see `design.md`'s `SchedulerCore` component and `research.md`). A separate retrospective design review (2026-08-19, unrelated to the original 2026-08-07 follow-ups) additionally found and fixed an `OPCode_schedAdmin` blocking-wait admission gap and extended `core.c`'s isolation-boundary test coverage to its `core_events.c`/`core_session.c`/`core_clock.c` siblings — see `research.md`'s "SUPERSEDES" notes. **scheduler-allocation** — lazy iterator-driven allocation architecture, parallel job support, session-level unsatisfiable resreq cache; requirements generated, `blocked_on: [scheduler-core]` — now unblocked. **scheduler-preemption** — SIGSTP-based victim selection + controller→worker pipeline; requirements generated, `blocked_on: [scheduler-core]` — now unblocked. The parent `scheduler-framework` spec is retained only for discovery-context provenance and is no longer in flight itself. Next: `/kiro-spec-design scheduler-allocation` and `/kiro-spec-design scheduler-preemption`.
|
||||
- ~~**ha-consistency-invariants** — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.~~ **COMPLETE 2026-07-13** — all four sub-specs (ha-commit-engine, ha-commit-barrier-rollout, ha-single-leader-role, ha-convergence-recovery) implemented; Criterion 2.9 (Facts-Apply invariant) satisfied via raft-apply-decoupling. No longer in flight.
|
||||
- ~~ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs `/xce-deploy-ha` cluster); sub-spec 4's E2E extends it~~ **task 5.4 CLOSED 2026-07-18** via `/kiro-validate-impl`'s live 10-round sequential soak (972/972 tests, zero loss). Spec is GO, no longer in flight.
|
||||
- ~~**libxce-failover** — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) `xce_request_with_retry()` primitive + adds multi-master rotation; classifies on the real controller status taxonomy `E_XCE_REDIRECT` / `E_XCE_NOMASTER` / `E_XCE_NOT_READY` / `E_XCE_STANDBY` (`errnox.h:22-25`); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, `controller.capnp:598-599`, just unpopulated in `join.c`). Verified E2E via `xce_test` (the original reproduction). Closes the integration gap left by `ha-config-bootstrap` Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).~~ **IMPLEMENTED, validate-impl GO 2026-07-18** (all 19 tasks were done since 2026-06-18; the `ready_for_implementation` flag was just never flipped — administrative staleness, not a real block). No longer in flight. `component-failover-adoption` (backlog, below) is the next layer on top of this.
|
||||
|
||||
## Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21)
|
||||
|
||||
**Goal:** evolve the election-only `src/lib/raft/raft.{c,h}` into a STRICT implementation of `raft/raft.tla` (Ongaro canonical) covering the log + commit half, keeping the pure callback-driven design so it stays independently verifiable via the deterministic simulator (`raft_sim.c`) against the TLA+ invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety). Motivation: **bug-851** — the log protocol is currently hand-rolled across `persist.c`/`repl.c`/`ha.c`/`ha_convergence.c` and violates the TLA+ log invariants. Gap analysis (six gaps G1–G6): `.kiro/specs/ha-term-safety/TLA-gap-analysis.md`. Implement in dependency order; each spec lands with simulator coverage. The library core stays pure/synchronous/deterministic — all I/O (async vtask, fan-out replication, the sync commit barrier, snapshot transport) lives in controller-side callback adapters. Preserve as policy (do NOT canonicalize away): PreVote, priority leadership, synchronous commit-barrier + persist-before-ACK durability, "controller never self-terminates".
|
||||
|
||||
- [x] raft-log-replication — *(specs generated + auto-approved 2026-06-21: 8 reqs, 24 sub-tasks, cross-spec review PASSED. Ready for /kiro-impl.)* Extend the library to OWN the replicated log. TLA+ actions: `ClientRequest`, `AppendEntries`, `HandleAppendEntriesRequest`, `UpdateTerm`-on-AppendEntries. Per-entry term in the library log model (closes **G1**); `AppendEntries` `logOk` = `prevLogTerm` match (**G2**); conflict branch deletes the divergent `(term,lsn)` suffix (**G3**); ANY higher-term RPC incl. AppendEntries/heartbeat advances `currentTerm` and steps down (**G5**). New pure callbacks: `on_persist_entry`, `on_get_entry_term(lsn)`, `on_send_append_entries(peer, prevLogIdx, prevLogTerm, entries, commitIdx)`, `on_truncate_from(lsn)`; controller adapters implement them over the existing WAL/repl transport. Simulator: drive AppendEntries across partitions/restarts, assert Log Matching. Dependencies: raft-library (election core). Closes bug-851 G1/G2/G3/G5.
|
||||
- [x] raft-commit-safety — *(specs generated + auto-approved 2026-06-21: 6 reqs, 18 sub-tasks, cross-spec review PASSED.)* `AdvanceCommitIndex` with the Figure-8 current-term rule (commit `Max(agreeIndexes)` only if `log[Max].term == currentTerm`) (**G4**); `RequestVote`/PreVote freshness becomes `(lastLogTerm, lastLogIndex)` instead of LSN-only (**G6**). New callback `on_commit_advanced(lsn)` feeds the existing synchronous commit barrier / durability (persist-before-ACK preserved as policy). Simulator: assert Leader Completeness. Dependencies: raft-log-replication. Closes bug-851 G4/G6.
|
||||
- [x] raft-sim-conformance — *(specs generated + auto-approved 2026-06-21: 8 reqs, 23 sub-tasks, cross-spec review PASSED.)* Extend `raft_sim.c` into a conformance/verification harness driving the library through `raft.tla`-action-equivalent transitions (randomized + targeted) and asserting the four Raft safety invariants; optionally trace-check against TLC output of `raft.tla`. The "use the verified spec to help" payoff: a regression oracle for the consensus core. Dependencies: raft-log-replication, raft-commit-safety. **IMPLEMENTED, validate-impl GO 2026-07-18** (23/23 tasks; 293/293 lib tests, zero ASAN findings; this is the spec this worktree/branch — `raft_tla` — is named for).
|
||||
|
||||
> **ha-term-safety** (`.kiro/specs/ha-term-safety/`) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the **config-change marker** (R7 — config changes as `(term,lsn)` log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live **bug-851 chaotic-failover regression**. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). **Scope: R1–R7 only.** The former R8 (config-upgrade full-snapshot consistency, bug-857) was **descoped 2026-06-21 to the independent `config-upgrade-snapshot` spec** (below), because it requires entity-bearing snapshots that do not exist yet. **IMPLEMENTED, validate-impl GO 2026-07-17** (20/20 tasks; closed bug-1445, a first-boot promotion-check race with job-array-single-wal-record's Live Apply Resource Fence, as an out-of-boundary follow-up fix).
|
||||
|
||||
> **raft-adapter-conformance** (`.kiro/specs/raft-adapter-conformance/`) — NEW 2026-07-29, sibling audit to the library-side conformance work above: a systematic, action-by-action pass over the controller's Raft **adapter** (the wire-level send/receive/reply code in `ha.c`/`repl.c`/`config_repl.c`) against `raft.tla`'s actions — distinct from `raft-sim-conformance`'s in-process library harness. Motivated by a live-soak finding that the controller's `WalAckReply` constructor hardcoded success (no failing reply ever reached the primary) plus a standby catch-up streaming-range self-deadlock, both already fixed (bug-1479) before this spec was seeded. This pass found and fixed two more instances of the same defect class: configuration-push rejects previously sent no reply at all (bug-1480), and a vote/PreVote endpoint failure arriving *after* connect routed through a legacy tally path instead of the production one (extending bug-1423's connect-time fix to that remaining timing window). Also records, as a single explicit decision, which `AppendEntries` mechanism is production-authoritative (`repl_broadcast` + `ha_stream_wal_to_standby`; the per-peer index-driven `raft_send_append_entries` stays dormant by documented design, not oversight). Requirements/design/tasks generated and approved, all 15 leaf tasks (1–7.3) checked; `src/controller/CLAUDE.md`'s Protocol Verification Checklist marks the corresponding invariants "post `raft-adapter-conformance`, landed 2026-07-29" — `spec.json` phase (`tasks-generated`) has not been flipped to reflect this, same administrative-staleness pattern seen elsewhere in this doc (cf. the former libxce-failover entry above). Dependencies: raft-log-replication, raft-commit-safety, ha-term-safety, raft-sim-conformance (adjacent/informational — this spec fixes the adapter layer those validate underneath).
|
||||
|
||||
## Raft Apply-Ordering & Hook Decoupling — NEW 2026-06-24 (`/kiro-discovery`); IMPLEMENTED — see `raft-apply-decoupling` below
|
||||
|
||||
**Goal (historical — achieved):** make the controller follow canonical Raft ordering — *pre-hooks → raft commit → apply → notify* — with the raft protocol step lifted OUT of the notification hook fan-out. At the time this family was opened, `persist_job_hook_fun` (an `AFTER_EVENT` hook) performed the WAL append + synchronous majority-commit barrier inline, and `create_job()` mutated in-memory state BEFORE that commit — the structural root of the bug-795/828/851/1060 divergence family. A failed/timed-out commit did not cleanly revert the optimistic mutation. Decision (user, 2026-06-24): **apply-after-commit (canonical)** — a single shared `apply_committed_entry()` seam mutates the state machine only after commit, reused by restart replay so online-apply and replay-apply are identical. Hooks demote to notification-only. Decomposed via Path E (2 new specs + touchpoints), delivered as `raft-apply-decoupling` (**IMPLEMENTED 2026-07-07**, see below). Current code reflects the fix: `persist_job_hook_fun` (`persist.c` ~6509-6553) now explicitly returns early for `WAL_MODE_WAIT` entries, with a comment noting job-submit durability is invoked directly by `handle_jobsubmit` — `persist_submit_job_payloads_and_commit()`, called strictly before any job object is constructed — never from this hook; `create_job()` is now defined at `job.c:48`.
|
||||
|
||||
### Async-drive prerequisites (surfaced 2026-06-25 by blocking audit; thread-role contract now in `ha-async-io.md`)
|
||||
|
||||
The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_handler` runs inline on iomgr instead of dispatching to the vtask pool, and delaymgr (the heartbeat driver) was the documented blocking sink. Corrected thread-role contract is now steering (`ha-async-io.md`, 2026-06-25; SUPERSEDES "delaymgr for blocking ops = Permitted"). Two separable prerequisites (full evidence: `.kiro/specs/raft-apply-decoupling/facts-in-raft.md` § Keystone + "Two distinct prerequisites" + P2 offload mechanism):
|
||||
|
||||
- **P1 — commit-barrier async** (the *commit* drive). Convert `repl_wait_majority_ack()` from blocking `pthread_cond_timedwait` (commit.c:359) to event-driven `resolve(lsn,outcome)` on the existing commit engine. Dissolves apply-self-deadlock (B), enables reply continuations (C), removes vtask-pool exhaustion. **Owned within `raft-apply-decoupling` as its keystone** (sequenced before its Phase A).
|
||||
- **P2 — drive hardening** (the *transport/timer* drive). Now its own spec **`raft-drive-async`** (brief: `.kiro/specs/raft-drive-async/brief.md`) — promoted because it threatens liveness independently. See the Specs list below.
|
||||
|
||||
### Specs (dependency order — Raft Drive & Apply Async family)
|
||||
|
||||
- [x] **raft-drive-async** — **IMPLEMENTED, validate-impl GO 2026-07-18** (21/21 tasks; the live sequential soak this validation ran surfaced a real crash, bug-1447 — a same-batch endpoint use-after-free in `io_worker`'s epoll dispatch loop, exposed by this spec's own faster reconnect cadence, see `ha-async-io.md`; root-caused, fixed, re-soaked clean, re-validated GO same day). Original description retained below for provenance. *(was: START FIRST — threatens failover liveness TODAY)* — bring the code into compliance with the corrected `ha-async-io.md` thread-role contract: dispatch `peer_ctrl_msg_handler` to the vtask pool (root-cause fix); snapshot send + config-push off iomgr → admission-capped slow lane (per-peer dedup + WAL coordination); delaymgr peer connect → bounded non-blocking + offloaded (reuse `xce_probe_config_tag` pattern); add a general non-IO vtask-submit path + admission reservation (extend `COMMIT_WAIT_RESERVED`). Verify with a **partition** (unreachable-not-killed) soak: heartbeats to survivors continue, no spurious election. Likely root of the bug-851 chaotic-failover family. Dependencies: none hard (builds on existing commit engine + vtask + the steering contract); `election-iomgr-refactor` is completed upstream (provides `delay_task_ms`/`on_connected`/non-blocking vote fan-out) — STANDALONE per 2026-06-25 scope-check (ADJACENT-SAME-AREA: it fixed startup blocking, this fixes runtime drive blocking). Brief + requirements + research + **design + tasks done**; phase=tasks-generated (tasks `approved: false`). **`/kiro-validate-design` GO 2026-06-25** — three review findings folded into design+tasks: (1) **deadlock-freedom + LivenessAdmission** — the documented invariant reversal (ack/`commit_on_ack` moves off iomgr to the vtask pool, `commit.c:16`) co-locates commit-barrier wakers+waiters; review found `COMMIT_WAIT_RESERVED` is **insufficient** because it counts only commit-waits while peer_ctrl also puts vote-grant fsync + config-ack waits on the pool (invisible to it). Fix: generalize the reservation to a **pool-level task-class admission in vtaskmgr** (`VTASK_LIVENESS`/`LIVE_BLOCK`/`REQ_BLOCK`; liveness never gated; `COMMIT_WAIT_RESERVED` folded in) — new **Phase-1 task 1.6** (dormant infra), hard prerequisite of 3.1; task 3.4 is the admission gate. Transient until `raft-apply-decoupling` P1 makes the barrier async; (2) **WAL-trim cursor-bound guard** added to task 3.2 (bug-1147, see Hardening below — moving checkpoint/trim onto a concurrent lane worsens the existing latent trim-vs-cursor race); (3) **in-process partition primitive** in `xce_test` as the portable primary for R7 (firewall DROP secondary) so the soak actually runs unprivileged. **Next: approve tasks (`/kiro-spec-tasks raft-drive-async`) → `/kiro-impl`; ship Phase 1 (bounded connect) first.**
|
||||
- [x] raft-apply-decoupling — decouple the raft protocol step from the hook fan-out (item #1); canonical pre-hooks → commit → apply → notify ordering (item #2); single `apply_committed_entry()` seam shared by online commit + restart replay; post-restart validation that committed log was correctly applied to materialized state (item #4); requirement + regression tests (item #5). **Contains P1 (commit-barrier async) as its keystone.** Commands (WAL_MODE_WAIT) apply-after-commit; **facts (WAL_MODE_RETAIN) routed through commit-then-apply too** — D1 RESOLVED (block-but-durable under quorum loss); formalized as `ha-consistency-invariants` criterion 2.9. Dependencies: raft-drive-async, raft-log-replication + raft-commit-safety (DONE/implemented), worker-rejoin-resync; coordinate with ha-term-safety. Briefs: `.kiro/specs/raft-apply-decoupling/brief.md` + `facts-in-raft.md`. **IMPLEMENTED 2026-07-07: 32/32 tasks, validate-impl GO** — 79/79 controller_unit_test groups (926 tests) green, live 3-node smoke boot verified, zero blocked tasks. Task 7.2 fixed bug-1416 (boot-time TOCTOU false APPLY_DIVERGENT fence). Task 7.4's own throughput benchmark found and fixed a real O(n²) apply-pump regression (56% throughput drop, superlinear scaling) via a segment-read cache on `persist_t`, 2 adversarial review rounds. Residual, non-blocking: design.md has 3 sections not yet reconciled with implementation deviations already self-documented in tasks.md's Implementation Notes (repl.c's planned `repl_submit_majority_commit()` was never built; apply pump wired via `commit_resolve_pending()` not `ha.c`'s `on_commit_advanced`; fact hooks collapse to generic `HOOK_DETAIL_JOB_UPDATE` not per-type); task 5.3 touched `ha.c` without a formal `_Boundary:_` exception tag (substantively justified, process-inconsistent only). Two out-of-scope findings flagged for follow-up: an intermittent client hang in the synchronous `jobsub` path (resembles but not confirmed as bug-1416's class), and `trim_wal_segments`'s retention floor has no awareness of the local apply pump's own lag (same family as an already-tracked standby-cursor gap).
|
||||
- [ ] worker-rejoin-resync — **PROMOTED to spec 2026-08-10** (`/kiro-discovery`; brief at `.kiro/specs/worker-rejoin-resync/brief.md`). **Elevated to HARD prereq for the facts half of raft-apply-decoupling 2026-06-25** (was backlog "designated backstop"). Worker re-reports running/finished job state on rejoin so facts lost in a failover window are recovered; re-reports flow through commit-then-apply idempotently (criterion 2.9). Today workers send only `hostattrs` on join (bug-396). Scoped narrowly to **worker** job-execution-fact resync only — apiserver/scheduler resync-on-reconnect is a different data shape (cluster/queue/job *list* state, not job *execution facts*) and stays out of scope (see the deferred "lightweight rejoin re-attach" backlog item below, still YAGNI). Dependencies: ha-consistency-invariants (commit barrier + idempotency guards) — both implemented, unblocked. Next: `/kiro-spec-requirements worker-rejoin-resync`.
|
||||
- [ ] job-array-single-wal-record — **PROMOTED from backlog 2026-06-24.** Persist/replicate/push a job-array submission as ONE Raft record (base spec + index range/count + per-index overrides) committed atomically through a single barrier; expand on apply (online + replay) and on the scheduler side; preserve per-element identity/addressing; idempotent re-apply (item #3). Dependencies: raft-apply-decoupling (HARD — builds on the shared apply seam), ha-commit-engine, ha-commit-barrier-rollout, ha-consistency-invariants. Brief: `.kiro/specs/job-array-single-wal-record/brief.md`.
|
||||
|
||||
### Existing Spec Updates (Raft Apply-Ordering phase)
|
||||
|
||||
- [ ] raft-sim-conformance — add apply-after-commit regression assertions (item #5): no apply on a failed/uncommitted entry; online-apply ≡ replay-apply for the same committed log; ordering invariant pre-hooks → commit → apply → notify. Revisit with `/kiro-spec-requirements raft-sim-conformance` once raft-apply-decoupling requirements land.
|
||||
- [ ] ha-term-safety — coordinate: raft-apply-decoupling restructures the same controller submit→commit path this umbrella wires. Sequence the apply-decoupling edits against ha-term-safety's adapter work (disjoint-but-adjacent regions) to avoid conflicting refactors.
|
||||
|
||||
### Direct Implementation Candidates (Raft Apply-Ordering phase)
|
||||
|
||||
- [ ] xce_test waldump/walcompare/snapdump inspectors (already a Hardening item below) double as the offline evidence for the item #4 restart-validation requirement — wire their nonzero-exit-on-divergence into the decoupling spec's E2E rather than as a separate spec.
|
||||
|
||||
## Specs (dependency order)
|
||||
|
||||
> `[x]` = spec generated (requirements/design/tasks approved, ready for implementation). Implementation tracked via each spec's tasks.md; implement in roadmap order (cross-spec review 2026-06-11: four specs touch ha_dispatch_gate / raft_cb_become_follower in disjoint-but-adjacent regions).
|
||||
|
||||
- [x] ha-commit-engine — commit.c per-LSN majority engine + quorum accounting fixes, wired behind job submit. Dependencies: none. (specs done 2026-06-11; **IMPLEMENTED 2026-06-12**: 12/12 tasks, validate-impl GO, 467 tests green; deferred follow-ups in its tasks.md §Implementation Notes)
|
||||
- [x] ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; **IMPLEMENTED 2026-06-13**: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
|
||||
- [x] ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
|
||||
- [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
|
||||
- [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.)
|
||||
|
||||
## Future Specs / Backlog (dependency order)
|
||||
|
||||
- [ ] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **Former HARD dependency `entity-bearing-snapshots` is CLEARED** — it shipped as spec `ha-entity-snapshots` (**IMPLEMENTATION-COMPLETE, validate-impl GO 2026-07-18**; see Future Specs/Backlog below), and `ha-term-safety` (the config-change marker it anchors on) is also implemented. `config-upgrade-snapshot`'s own `spec.json` now has `blocked_on: []`. Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). **Ready to promote to design.**
|
||||
- [ ] component-failover-adoption — **recorded 2026-06-16; depends on libxce-failover (HARD).** Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link `libxce_dep`; apiserver uses it via cgo, `src/frontend/apiserver/backend/clib.go`), but two per-component integration items remain: (1) each component must hand libxce the **full master list** instead of a single endpoint (today even the CLI uses only `conf->masters[0]`, `client.c:122`); (2) bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons (worker `controller_connector.c`, scheduler `handler.c`) this must be verified **liveness-safe**, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. (`xce_test` is NOT here — it is the in-spec verification client of libxce-failover.) Promote via `/kiro-spec-init component-failover-adoption` after libxce-failover lands.
|
||||
- [x] ~~worker-rejoin-resync~~ — **PROMOTED to spec 2026-08-10, see In Flight (Raft Apply-Ordering family) above** for current status and brief. No longer tracked here.
|
||||
- [ ] **admin-config-tool** — **PROMOTED from backlog to spec 2026-07-06** (`/kiro-discovery`; brief at `.kiro/specs/admin-config-tool/brief.md`). **NEW: `xceadmin` Python CLI** for cluster operators — builds and deploys cluster configuration via direct TCP+TLS capnp wire to controller (NOT through apiserver). Two subcommands: **(a) `config build`** — directory scan: parse `configs/{hosts,queues,resources,users}.yaml`, validate against JSON schemas (`configs/schema/*.schema.json`), serialize `ClusterConfig` capnp struct to `cluster.bin` with SHA-256; **(b) `config push`** — zero-config dev (default `localhost:9991`, same as xce_test), single `--controller` flag for production, auto-discovers peers + primary from controller reply, sends `OP_upgradeConfig`, displays `{versionTag,ackedCount,error}`. **Single dependency: `controller.capnp`.** Pure Python (pycapnp + yaml + jsonschema + click), open-source ready (`pip install`). The tool is the client-side trigger for `config-upgrade-snapshot`'s controller-side path. MUST validate against reserved name `"lost_and_found"` for queue/host (defence-in-depth). xcecli comparison: `xcecli` = user tool (HTTP/HTTPS → apiserver → CGO → controller, JWT auth); `xceadmin` = operator tool (direct TCP+TLS → controller, zero-config, same trust model as controller peers). Shared future: `xcecommon` Python package with capnp schema + wire protocol reusable by both tools. Dependencies: `controller.capnp` schema only. **Blocks: nothing — ships independently.** Controller-side upgrade snapshot path (`config-upgrade-snapshot`) was blocked on `entity-bearing-snapshots`; that blocker is now cleared (see Future Specs/Backlog above — `ha-entity-snapshots` implemented, `config-upgrade-snapshot` `blocked_on: []`). Next: `/kiro-spec-requirements admin-config-tool`.
|
||||
- [ ] **component-rejoin-resync** — **PROMOTED to spec 2026-08-10** (`/kiro-discovery`; brief at `.kiro/specs/component-rejoin-resync/brief.md`) from the "lightweight rejoin re-attach" item below. **apiserver + scheduler** reconnect/rejoin resync efficiency — today each does a full re-join and receives full state again (whole job list, queue/resource/config state), not a delta; for a large cluster this is sizable but not yet a measured problem. Sibling of `worker-rejoin-resync` — disjoint data (list/config state vs. job-execution facts), NOT merged into that spec. Not started: first step is investigation (is reconnect cost actually a measured problem at realistic cluster sizes?) before committing to any delta/session-token mechanism — original YAGNI discipline still applies, this spec is the promoted vehicle for that investigation, not a rubber-stamped design. Original decision (libxce-failover design review) retained below for provenance. Dependencies: libxce-failover (rejoin hook — implemented), component-failover-adoption (daemon rejoin content — in progress, sequence alongside/after). Next: `/kiro-spec-requirements component-rejoin-resync` when ready to begin (not immediate).
|
||||
- [x] ~~lightweight rejoin re-attach (failover optimization)~~ — **recorded 2026-06-16; deferred (YAGNI)**; apiserver/scheduler half **PROMOTED to `component-rejoin-resync` 2026-08-10** (see above); worker half already promoted to `worker-rejoin-resync`. No longer tracked as a bare backlog item — provenance only. Original text: on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale.
|
||||
- [x] job-array-single-wal-record — **IMPLEMENTED, validate-impl GO 2026-07-17** (reopened once 2026-07-16 for three user-directed residual gaps — bug-1443 bulk-insert partial-visibility, task 4.2's boot-replay fence gap, bug-1444 replay_wal array-blind tally — all closed, re-validated fresh GO). **PROMOTED to a spec 2026-06-24** (`/kiro-discovery`; brief at `.kiro/specs/job-array-single-wal-record/brief.md`; now tracked under the *Raft Apply-Ordering & Hook Decoupling* phase above, sequenced after raft-apply-decoupling). Original note retained for provenance: Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready.
|
||||
- [ ] bulk job-array query/cancel — **recorded 2026-07-13, deferred out of `job-array-single-wal-record` scope by design.** A single client operation to query or cancel an entire array (not one element at a time) is explicitly out of scope for `job-array-single-wal-record` — see that spec's Boundary Context. Reason: array elements can be in heterogeneous states at request time (done/exited elements need no action; running elements need a worker kill signal; pending elements need a direct state transition to cancelled without ever running), which is a materially different bulk-operation problem from atomic array *submission* and needs its own requirements (partial-failure semantics, atomicity vs best-effort per element, etc.) rather than being folded into the submission spec. Dependencies: `job-array-single-wal-record` (per-element identity/addressing this would operate over). Promote via `/kiro-discovery` once job-array-single-wal-record ships.
|
||||
- [x] ~~WAL group-commit batching~~ — **recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal).** The `/kiro-validate-design` review surfaced that the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible and not covered by `job-array-single-wal-record` — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): **do group-commit now** — repurpose `fsync_worker` from a timer into a demand-driven flusher (batch one `fdatasync` across pending appends, advance `durable_lsn`, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
|
||||
- [ ] primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
|
||||
- [ ] read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
|
||||
- [x] entity-bearing snapshots ("Task 11" format) — **PROMOTED to spec `ha-entity-snapshots` (created 2026-07-01); IMPLEMENTED, validate-impl GO 2026-07-18** (21/21 tasks 1.1-8.1; 962 controller tests green including all 9 `test_persist_snapshot_entities.c` cases; live `--force-snapshot` soak: committed_divergence=0, committed_job_loss=0, confirmed real "HA: snapshot" install log line). `write_snapshot()` (`persist.c` ~2304-2372) now collects and serializes job/queue/host entity frames (`collect_job_frames`/`collect_queue_frames`/`collect_host_frames`) ahead of the header — no longer a bare 16-byte LSN sentinel. Original backlog note retained for provenance: real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs was already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. **Downstream: `config-upgrade-snapshot`'s former HARD dependency on this is now cleared** (`blocked_on: []`) — see that item above. **Also strategy B for catch-up of a far-behind / empty-rejoin standby (cursor=0)** is now live: the standby-side snapshot-install/apply path is wired (`ha.c` ~3112-3147 wire-receive path task 6.1; `persist.c` ~4277-4285 local boot recovery task 8.1), with a decision path that requests a snapshot transfer when a standby's cursor falls below the leader's oldest retained WAL LSN (`ha.c:4413`). The bug-1147 WAL-trim cursor-bound guard (see Hardening below) is implemented in production, no longer just a stopgap.
|
||||
|
||||
## Hardening / Direct-Implementation Items
|
||||
|
||||
Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.
|
||||
|
||||
- [x] ~~eliminate src/frontend/Makefile, consolidate Go builds onto meson~~ — **PROMOTED to spec 2026-08-27** (`/kiro-discovery`; brief at `.kiro/specs/eliminate-frontend-makefile/brief.md`). Investigation (recorded 2026-08-21, from `dirty-content-fingerprint` spec's task 4.3) confirmed the risk is real and independently corroborated: `build-server` is separately confirmed BROKEN today (bug-1597, bare `go build` with no CGO flags, `fatal error: client.h: No such file or directory`; `src/docs/deployment.md:243-247` already tells developers to use `/xce-build apiserver` instead). Blast radius is narrower than originally framed: `/xce-build` already builds both binaries via meson exclusively (`build-cli`/`build-server` are dead weight from that path today); only 2 live call sites (`xce-build.sh:240,287`) depend on `make test`, with an already-proven direct-`go test` replacement 3 lines away in the same file. One genuine gap found: no meson `test()` exists for the CGO backend suite (`test-backend`) — that must be added before the Makefile can be safely removed, or backend CGO coverage is lost, not just inconvenienced. Next: `/kiro-spec-init eliminate-frontend-makefile`.
|
||||
- [ ] **job_ctrl direct-mutate path — migrate to commit-then-apply** — **recorded 2026-07-13, investigated 2026-08-19 (finding: real risk, wider scope than originally framed).** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes). The 2026-08-19 investigation found the original framing wrong in two ways: (1) **NOT the only remaining call site** — `on_job_stop`/`on_job_resume`/`on_job_ckpt` (`job.c:2067-2222`) share the identical unconverted mutate-then-notify-then-WAL-append pattern, fed by ordinary worker/scheduler traffic, not "rare, admin-only" as originally described; (2) **blast radius (a) is confirmed real, not theoretical** — the pre-commit mutation is synchronously pushed live to BOTH the scheduler (`sched_job_update_fun`) AND every connected apiserver/client (`apiserver_job_update_fun`) before the commit barrier resolves; on barrier timeout the admin gets a failure reply but `job->job.state` is never rolled back and no retraction is ever sent — matching the bug-795/828/851/1060 shape. Live-apply vs replay-apply (b) is narrower than feared for `job_ctrl` specifically (the `allocs != NULL` gate mutually excludes it from the converted deferred-apply fact types), but the investigation surfaced a SEPARATE bug underpinning that safety argument: `job_ctrl` (`job.c:999`) holds only a read lock while mutating job state — every other mutator in this file takes a write lock — so two concurrent admin commands on the same job aren't mutually excluded, undermining the "same lock, no staleness window" claim the replay-safety argument depends on (already flagged by a `raft-apply-decoupling` task 2.1 reviewer, never fixed). **Recommendation**: promote to a small follow-up spec covering all four call sites together (bounded, well-precedented fix — same shape as the fact-receipt conversion `raft-apply-decoupling` already built for job submit); the lock-type bug is independently fixable and low-risk regardless of when/whether the larger migration happens. Not yet promoted — user decision 2026-08-19 to correct scope/framing here first, promote later.
|
||||
- [x] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — **IMPLEMENTED** as part of `raft-drive-async` task 3.2 (`raft-drive-async` is IMPLEMENTED, validate-impl GO 2026-07-18; see above). `trim_wal_segments` (`persist.c` ~2597-2644) now computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` — folding in active-standby `acked_lsn` and disconnected-slot `last_disconnected_lsn` — and only unlinks segments below that floor; the code comment at `persist.c:2634` explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that `entity-bearing snapshots` (strategy B) has landed as `ha-entity-snapshots` (see above), with a live decision path (`ha.c:4413`, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060.
|
||||
- [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_
|
||||
- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`.
|
||||
- [ ] validate_host() real authorization — currently a stub returning success for all peers (`join.c:13-16`, `return 1;` unconditionally). **`CONF_ENABLE_UPGRADE_CONFIG` is already `1` in the standard build** (`config_repl.h:27-28`, no build-system override found) — the precondition this item was originally framed to guard against has already been crossed; `OP_upgradeConfig` (and `OPCode_schedAdmin`, added 2026-08-02, `scheduler-core`) are live and unauthenticated TODAY, not a future risk to prevent. Minimum fix: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` and `OPCode_schedAdmin` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.**
|
||||
- [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ **Part of the broader `peer-authentication-hardening` item above.**
|
||||
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
|
||||
- [x] ~~OP_upgradeConfig standby redirect verification~~ — **RESOLVED 2026-08-19, then the underlying gap itself CLOSED 2026-08-26 (bug-1700/1701).** As of 2026-08-19, a standby did NOT return the primary's address for `OPCode_upgradeConfig` — `ha_dispatch_gate` rejected it before the handler's own dead defensive check ever ran, replying flat `status=-XCE_ERR_STANDBY, data_which=nodata` with no redirect fields; `fill_redirect_reply()` (`join.c:27`) was never called for this opcode, a documented, deliberate feature gap (`admin-config-tool` design.md's "Revalidation Triggers"). That gap is now closed: `cluster.c`'s generic dispatch-gate rejection path calls `fill_redirect_reply()` on every `-XCE_ERR_STANDBY` rejection (bug-1700/1701, gate-level fix, not opcode-specific), and a standby pushing `config push` now redirects to the primary and succeeds. Coverage: client-side redirect-following remains unit-tested against mocks (`test_discovery.py`, `test_config_push_mock.py::test_push_config_follows_one_redirect_to_primary`); the real server-side behavior is live-verified against a real 3-node cluster (`src/xceadmin/tests/test_e2e_live_ha_push.py::test_live_config_push_against_standby_redirects_and_succeeds`, renamed from `..._returns_standby_rejection_not_redirect` to match the now-correct outcome).
|
||||
- [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
|
||||
- [ ] xce_test WAL/snapshot inspection tooling — extend the offline inspector family in `xce_test` (sibling of the shipped `waldump`, `src/controller/testsuite/waldump.{c,h}` + `cmd_waldump`). Two remaining commands: **(a) `snapdump <snap_dir|snap_file>`** — decode the `XCESNAP\0` header (8-byte LSN; `persist.c:1266` `load_snapshot`) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (`.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md`): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. **(b) `walcompare <dirA> <dirB>`** — diff two nodes' WALs by `lsn → crc/jobid` to flag divergence/prefix-gaps between members instantly. Reuse the `waldump.c` reader/decoder; same zero-alloc capnp jobid decode + `c_capnproto_dep`/`zlib_dep` wiring already added to `xce_test_deps`. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. _(recorded 2026-06-20; `waldump` already done.)_
|
||||
- [x] ~~xce_test shell command support~~ — **IMPLEMENTED** (task 2.1, spec `xce-test-e2e-primitives`, closed 2026-06-19; stale entry corrected 2026-08-19). `preprocess_and_run()` (`src/controller/testsuite/test.c:950`) detects a leading `!` and runs the remainder via `system()`; the shell command's exit status becomes `g_last_status` (the `$?` auto-variable) and a non-zero status sets `g_session_failed`, aborting the script the same way any other failing command does — `system()`'s child inherits the parent's stdout/stderr fds, so output lands in whatever the invoking `xce_test` session's own output is already redirected to. Works in every REPL context (no `leave_ctx()`/disconnect), including mid-session inside `worker`/`client` contexts — used this session to fix bug-1579 (resource-value-verification's nested controller-check pattern). No parser/harness-layer change remains to do.
|
||||
- [x] scheduler framework enhancement — **PROMOTED to spec 2026-07-06, then DECOMPOSED 2026-07-27** into `scheduler-core` (**IMPLEMENTATION-COMPLETE**), `scheduler-allocation`, and `scheduler-preemption` — see **In Flight** above for current status. Scope notes from the original brief still apply: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). No longer tracked here.
|
||||
- [→] **perf-tuning-extraction** — **PROMOTED to spec 2026-07-20** (`clusterconfig-schema-split`, `.kiro/specs/clusterconfig-schema-split/`). Originally recorded 2026-07-06 from admin-config-tool discovery as a documentation/audit item; the audit (2026-07-20) found the fields are load-bearing enough to need real code changes, not just a doc pass — `candidates`/`vcpus`/hash-tuning fields (`jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`) are already dead weight in the `ClusterConfig` capnp blob (superseded by `--peers`/`--vcpus` CLI flags or never read into runtime state — see `src/docs/cluster-config-split-analysis.md`). **`port` is NOT dead** — a second review pass (2026-07-20) found `cluster_open_listener` (`cluster.c:1216-1258`) falls back to the blob's `port` field, and single-node mode (no `--peers` flag at all, `main.c:293-294`) depends on that fallback entirely; the spec now requires a replacement port-sourcing mechanism before the field can be removed (`clusterconfig-schema-split` Requirement 1.7), not a bare deletion. `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness, NOT the HA/Raft election heartbeat) were investigated and deliberately kept IN `ClusterConfig` for operational uniformity despite passing the independence rule — see `.kiro/steering/config-design.md`. `fetchJobInterval`/`netioTimeout` left unresolved, flagged out-of-scope in the new spec for a future pass.
|
||||
- [ ] **Raft/HA heartbeat interval+timeout as CLI parameters** — **NEW 2026-07-20 from clusterconfig-schema-split discovery.** The controller-to-controller HA/Raft election heartbeat (`ha->config.heartbeat_interval_ms`/`heartbeat_timeout_ms` — the one where a cross-node mismatch risks spurious failover) is 100% hardcoded today (`HA_DEFAULT_HEARTBEAT_INTERVAL_MS`/`HA_DEFAULT_HEARTBEAT_TIMEOUT_MS`, `cluster.c:1131`) — there is no `ClusterConfig` field for it, and per user decision (2026-07-20) there should never be one: these values must be available to a controller BEFORE it has validated/received cluster configuration (a controller needs its own election-timing parameters to even participate in the election that delivers config in the first place — a bootstrap/chicken-egg constraint, not an independence/frequency judgment call). Target: `--heartbeat-interval-ms`/`--heartbeat-timeout-ms` CLI flags following the exact `--vcpus` pattern (`OPT_*` + `struct arguments` + `myconf_t` field, `0` sentinel = use hardcoded default). Kept hardcoded for now; this is a future item, not blocking `clusterconfig-schema-split`.
|
||||
- [ ] **resource-distribution-mapping** — **created 2026-07-23, never added here (found 2026-08-19, itself a small doc gap).** Physical resources have no "configured total vs. currently-consumed" concept anywhere in this codebase, live or dead — `resources.yaml`'s `resourcemap` field (which expresses exactly this: a value shared across N hosts) is deliberately dropped by `xceadmin config build` (`build.py:41-43`), and the matching `controller.capnp` fields are commented out (`controller.capnp:208-209`). `mark_host_res()` (`worker.c`) actively rejects any resource flagged `XCE_RESF_SHARED` today, so "shared pool" and "per-host value" are mutually exclusive by construction in the live system right now. A working total/current pooling mechanism DOES exist in the scheduler (`resource.c`'s `sched_check_shared_satisfy_()`), but it exclusively serves job-limit virtual resources (queue `MAX_RUNNING`-style), never physical shared resources — architecturally reusable, never connected. This spec (research.md done, 3 scoped options, requirements.md never generated — sitting at the decision point) is the existing owner of closing this gap; surfaced again 2026-08-19 when scheduler-core's REPL `info resource` command couldn't show a total for a GLOBAL/SHARED resource because there genuinely isn't one to show. Next: pick a scope option in research.md, then `/kiro-spec-requirements resource-distribution-mapping`.
|
||||
- [ ] **scheduler-sim-diagnostics** — **NEW 2026-08-10 (`/kiro-discovery`).** Open-ended, continuously-extended spec for scheduler simulation-mode debug/troubleshoot capability — added to over time as new investigation needs surface from discussion or from other in-flight scheduler specs (`scheduler-allocation`, `scheduler-preemption`). Extends `scheduler-core`'s REPL (`info job/queue/worker/resource/summary`, breakpoints, snapshots). Two seed capabilities identified so far: (1) **limit inspection** — `xce_limit_t`/`xce_vres_t` config + live usage currently live only in the `limit` plugin's private kvmgr tables, invisible to the REPL; needs a new `info limit <name|all>` command + accessor plumbing. (2) **decision-reasoning trace** — `xce_pendreason_t` is captured on pend decisions but never printed by the REPL, and dispatch decisions have no reasoning trail at all (no record of which limit/resource checks passed) — needs new instrumentation in the allocation path (`resource.c`'s `sched_try_alloc_on_host_`, `plugin.c`'s `limit_prepare_allocation`) plus a REPL command to surface it. Explicitly simulation-mode only (no live/production controller-connected diagnostics) and read-only (no state mutation) per user decision 2026-08-10. Brief: `.kiro/specs/scheduler-sim-diagnostics/brief.md`. Not started — requirements to be added per-capability, not fully speced upfront. Next: `/kiro-spec-init scheduler-sim-diagnostics` when ready to begin.
|
||||
- [ ] **xcecd_virtual concurrent-job support (MXJ > 1)** — **NEW 2026-08-19, recorded during bug-1589 investigation** (`.wolf/buglog.json`). Live testing found that when a job array (multiple elements) is dispatched to a single `xcecd_virtual` worker in quick succession, only the first element's start ever gets confirmed back to the controller (`on_job_start`) — the rest are dispatched but never confirmed. Root-caused (or being root-caused — see bug-1589 for the confirmed mechanism once the in-flight investigation lands) to `xcecd_virtual`'s job-simulation path: unlike the real `xcecd` binary, which launches an actual dbus-managed worker process per job (tracked by real pid, real IPC status reports), `xcecd_virtual` (`src/worker/src/xcecd.c`'s `xcecd_process_command()`, `VIRTUAL_CLUSTER` branch) never launches a real process at all — it assigns a bare incrementing fake pid (`static int virtual_pid`, `xcecd.c:27`) and simulates job execution via a sleep-timer mechanism (`xcecd_worker_utils.h`'s `VIRTUAL_CLUSTER`-only `xcecd_worker_t` fields `start_time`/`sleep_time`/`killed_signal`, `check_worker_sleep_time_elapsed()`). The immediate bug-1589 fix (once landed) addresses the specific reported symptom; this item is the broader follow-up: **properly support N concurrent simulated jobs per `xcecd_virtual` worker**, since MXJ (max jobs per host) > 1 is a legitimate, common configuration to want to exercise in virtual-cluster-mode testing/CI (this project's own dev workflow relies on `xcecd_virtual` for HA soak tests, resource-value verification, and now scheduler-core's live validation — several `.wolf/buglog.json` entries this session alone used it), not just a single-job-at-a-time edge case. Scope sketch (to refine once bug-1589's root cause is fully confirmed): audit every VIRTUAL_CLUSTER-only code path in `src/worker/` (`xcecd.c`, `main.c`, `xcecd_worker_utils.h`/`.c`, `process_worker_manager.c`) for single-job assumptions (a lone poll/early-return that only processes one pending simulated job per tick, a shared mutable scratch struct reused across concurrent jobs, etc.) and generalize each to iterate the full set of in-flight simulated jobs, matching the pattern the real-process path (`process_manager.c`'s `process_info_t manager->processes[]`, a proper per-pid array) already establishes correctly. Dependencies: bug-1589 (must land first — its root-cause trace is the concrete map of what needs generalizing). Promote via `/kiro-discovery` or `/kiro-spec-init` once bug-1589 is closed and the exact mechanism is confirmed.
|
||||
|
||||
## Superseded / Absorbed Items
|
||||
|
||||
- ~~Quorum-ACK gate before config restart~~ _(FUTURE_WORK.md §2)_ — **largely superseded by ha-consistency-invariants Requirement 5**: configuration changes now require majority acknowledgment before being confirmed/committed, and fail to the requester on timeout (no silent rollback when standbys are offline). Residual to verify in that spec's design: restart sequencing after a committed config change (don't restart the primary before the commit barrier passes).
|
||||
- ~~Lost-and-found keyword validation~~ _(FUTURE_WORK.md §8)_ — folded into the admin-config-tool spec (brief: `.kiro/specs/admin-config-tool/brief.md`).
|
||||
- ~~admin-config-tool backlog item~~ _(FUTURE_WORK.md §1 + §8, 2026-05-12)_ — **absorbed into `admin-config-tool` spec 2026-07-06** (`/kiro-discovery`). The one-paragraph backlog sketch is superseded by the full brief with architecture decisions, decomposition, and boundary contracts at `.kiro/specs/admin-config-tool/brief.md`.
|
||||
|
||||
## Notes
|
||||
|
||||
- Backlog items here are seeds for `/kiro-discovery` / `/kiro-spec-init`; promote an item by creating its spec under `.kiro/specs/`.
|
||||
- Provenance: items marked _(FUTURE_WORK.md §N)_ originate from the 2026-05-12 config-replication-design.md review, merged here 2026-06-10.
|
||||
- **2026-07-06 admin-config-tool discovery** (`/kiro-discovery`, Path E mixed decomposition):
|
||||
|
||||
| # | Workstream | Type | Language | Dependencies |
|
||||
|---|-----------|------|----------|--------------|
|
||||
| 1 | **admin-config-tool** | NEW spec | Python | `controller.capnp` only |
|
||||
| 2 | **config-upgrade-snapshot** | EXISTING spec extension | C | entity-bearing-snapshots (blocked upstream) |
|
||||
| 3 | **perf-tuning-extraction** | DIRECT work | doc | None |
|
||||
|
||||
**Key architecture decisions:** Python (future open-source, shared capnp infra), pure Python wire (no CGO/libxce), direct TCP+TLS to controller (not apiserver), single `controller.capnp` dependency, `xceadmin` separate from `xcecli` (user=REST, admin=TLS), auto primary detection via peer list + redirect. Full details: `.kiro/specs/admin-config-tool/brief.md`.
|
||||
- **2026-07-06 scheduler-framework discovery** (`/kiro-discovery`, Path C single-scope):
|
||||
|
||||
**Key architecture decisions:** Iterator-driven allocation (lazy plan iterator + host iterator composition), configurable job picker (plugin name from config, not hardcoded), combinatorial parallel job allocation (1host×N → 2hosts×⌈N/2⌉ → … → Nhhosts×1 sorted by preference), session-level unsatisfiable resreq fingerprint cache. Scope narrowed from original backlog seed: commit barrier wiring deferred, new job selection policies deferred (framework enables them), GPU/affinity allocation deferred (pluggable via allocator interface). Alternatives documented in brief.md (Approach A: incremental/rejected, Approach C: full framework/deferred). Full details: `.kiro/specs/scheduler-framework/brief.md`.
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
# Scheduler Driver Equivalence
|
||||
|
||||
## Philosophy
|
||||
|
||||
`scheduler-core` split the scheduling engine into a mode-agnostic **core** state machine (`core.c`/`core_events.c`/`core_session.c`, `xce_sched_core_t`) driven by two interchangeable **drivers**: `driver.c` (controller-connected, capnp-driven, live) and `sim.c`/`sim_loop.c` (standalone, `.replay`-file-driven). The entire value of simulation/replay mode rests on one premise: replaying the exact recording of a normal-mode run reproduces what actually happened, so an operator can debug and tune scheduling against recorded production data and trust the result. `src/scheduler/CLAUDE.md` states this as "one core, two consumers" — this file makes it an enforced constraint, not a description.
|
||||
|
||||
## DESIGN INVARIANT: State equivalence, not just decision equivalence
|
||||
|
||||
**Replaying the exact recording of a normal-mode run MUST reproduce identical scheduling decisions AND identical resource-accounting state** (`w->resInUse`, `core->sharedInUse`, `job->accounts`) **at every point in the replay** — not just a matching final decision sequence.
|
||||
|
||||
Decision-only equivalence is an insufficient test of this invariant. A fixture with no resource contention (every job fits regardless of what `resInUse` says) can produce identical decisions between normal and sim mode while the underlying accounting state has silently diverged — because nothing in that fixture ever depends on `resInUse` being correct. The divergence only becomes decision-visible under contention, which is exactly the scenario most replay-based debugging exists to investigate. **Treat accounting-state parity as load-bearing, not incidental to decision parity.**
|
||||
|
||||
This is a stricter, broader sibling of the REPL-vs-batch-sim invariant (`--interactive` and batch `--sim` both drive `sched_sim_driver_run()` gated only by `sim->interactive`, proven decision-equivalent by construction and enforced by `test_repl_equivalence.c`). That invariant is solid — both paths share the exact same driver code. Normal-mode-vs-sim-mode equivalence is weaker: they are *separate* driver implementations that must each independently uphold the same per-round contract around the shared core, and nothing mechanically forces them to stay in sync when one changes.
|
||||
|
||||
## Why a driver can silently violate this
|
||||
|
||||
The core deliberately does not own accounting. `xce_core_begin_session()`'s resval reset zeroes `resInUse` every session; re-summing it from currently-running jobs (`sched_account_jobs()`) is documented as "real, load-bearing production behavior... `xce_scheduler_t*`-coupled and therefore outside this pure-core module's reach" (`core.h`) — i.e. each driver must call it itself, in the gap between `xce_core_begin_session()` and `xce_core_signal_go()`. `driver.c`'s `xce_sched_driver_run_normal_mode_round()` does this correctly. Nothing forces a second driver implementation to remember to.
|
||||
|
||||
Similarly, a `ReplaySession` carries a `decisions` list — the recorded decisions from the original live run — decoded every round via `decode_ReplaySession_ptr()`. There is no structural guarantee a driver applies that data (reconstructing `job->allocs`/`job->accounts` for a job already resolved in the recording) rather than only using it for optional comparison.
|
||||
|
||||
## Known violations (evidence trail)
|
||||
|
||||
- **bug-1613** (confirmed): `sched_account_jobs()` is called by `driver.c` but never by `sim_loop.c`, which still uses the older monolithic `xce_core_go()` with no interleaving point for it. Compounding this, `sched_account_jobs()`/`sched_charge_account()`/`sched_update_account()`/`sched_update_res_accounts()` are signatured on `xce_scheduler_t*` (the normal-mode-only driver struct) even though their bodies only ever touch `sched->core` — `xce_sched_sim_driver_t` deliberately has no `xce_scheduler_t*` field (Req 5 AC 17, simulation-isolation boundary), so these functions cannot even be called from the sim driver today without a signature change. `sched_resolve_job_picker()` went through this exact `xce_scheduler_t*` → `xce_sched_core_t*` relocation already (scheduler.h documents it) — the precedent to follow, not a new pattern to invent.
|
||||
- **bug-1611** (suspected same root cause): `sim_loop.c` only reads `decoded_session->decisions` when `sim->verify` is set, purely to compare against freshly-drained live decisions. It is never applied to reconstruct state for a job the live picker in this replay isn't independently re-deciding. This plausibly explains both bug-1611 (no `DISPATCH` decision ever visible for such jobs) and the accounting gap in bug-1613 (their usage is never counted, not even transiently) as one underlying gap, not two.
|
||||
- Both are tracked in `.wolf/buglog.json`; a dedicated fix spec is scoped separately (see `.kiro/specs/` — do not fix inline as a drive-by patch, per the testing discipline below).
|
||||
|
||||
## Testing discipline
|
||||
|
||||
Any change touching resource accounting (`resource.c`), core event application (`core_events.c`), or driver session-boundary handling (`driver.c`, `sim_loop.c`, `core_session.c`) must be validated against **a resource-contention fixture** — multiple jobs competing for capacity insufficient for all of them — not only a fixture where every job fits. A no-contention fixture cannot distinguish "accounting is correct" from "accounting is wrong but nothing currently reads it." Existing coverage in this space (a normal-mode-recording-vs-simulation-replay decision-identity test) evidently does not exercise contention, or bug-1613 would have failed it.
|
||||
|
||||
When adding or modifying driver-specific per-round logic in either `driver.c` or `sim_loop.c`, check the other driver for the equivalent step before assuming none is needed — the two are not required to look identical, but any core-derived state (`resInUse`, `sharedInUse`, `job->accounts`, decision counts) they produce from equivalent input MUST match.
|
||||
|
||||
## See Also
|
||||
|
||||
- `src/scheduler/CLAUDE.md` — architecture overview; "Drivers" section points here for the equivalence contract.
|
||||
- `.kiro/specs/scheduler-core/` — original driver-split design and requirements.
|
||||
|
|
@ -1,71 +0,0 @@
|
|||
# Security Standards
|
||||
|
||||
## Philosophy
|
||||
|
||||
Defense in depth, least privilege, fail closed. Authentication verifies *who*; authorization verifies *what they can do*. Centralize both — never duplicate permission checks across handlers.
|
||||
|
||||
## RBAC Enforcement
|
||||
|
||||
All protected routes use `RequirePermission` middleware attached at registration time:
|
||||
|
||||
```go
|
||||
// In router/routers.go
|
||||
hostsGroup.GET("", RequirePermission("host:view"), handler)
|
||||
hostsGroup.POST("/control", RequirePermission("host:control"), handler)
|
||||
```
|
||||
|
||||
`RequirePermission` in `controllers/rbac_check.go` resolves the user from the JWT, looks up their effective permissions (including inherited role permissions), and aborts with 403 if the required permission is absent.
|
||||
|
||||
**cluster_admin** bypasses all permission checks via early return — do not add new bypass paths outside this single mechanism.
|
||||
|
||||
**Known gap (not yet scoped): runtime RBAC mutations are not persisted.** `store.Roles`/`store.UserRoles` (`store/stores.go`) are loaded from `config/rbac.yaml` at startup via `LoadRBACConfig()`, but are also mutated at runtime with no write-back to disk or any DB — `RoleController.AddPermissionsForRole` (`controllers/roles.go`), and `store.AddRoleForUser`/`DeleteRoleForUser` (called from `AuthController.Logon`'s auto-role-assignment and the user-role-assignment endpoints in `controllers/users.go`). On apiserver restart, any role/permission granted or user/role assignment made via API since the last startup silently reverts to whatever `rbac.yaml` says, with no error or audit trail of the loss. This is the same save/restore field-completeness failure shape documented in `persistence.md` (state that "recovers correctly" while silently dropping data it depends on), just in the apiserver's in-memory RBAC store rather than a WAL/snapshot. Flagged during the `scheduler-snapshot-job-allocation` cross-component persistence survey (2026-08-27); intentionally deferred rather than fixed inline — needs its own scoped design (write-through to `rbac.yaml`? a small embedded store? explicit "runtime-only, does not survive restart" documented as accepted scope?) rather than a quick patch.
|
||||
|
||||
## Permission String Format
|
||||
|
||||
`"action:scope"` — e.g., `"queue:submit"`, `"host:view"`, `"queue:control"`.
|
||||
|
||||
Scopes: `all` (unrestricted), `self` (own resources), `cluster`, `hostgroup`, `self_ug`, `self_hg`.
|
||||
|
||||
Permissions are defined in `config/rbac.yaml`. Roles support inheritance via `includes:` — resolution is recursive. Add new permissions to the YAML first, then wire them onto routes.
|
||||
|
||||
## JWT Authentication
|
||||
|
||||
Token flow:
|
||||
1. `POST /auth/logon` → validates credentials via configured auth backend → issues JWT with `{username, IP, expiry}` claims
|
||||
2. Every subsequent request: `Authorization: Bearer <token>` header
|
||||
3. `middleware/validatehandler.go` strips the prefix, parses with golang-jwt, **validates the IP claim against the request IP**
|
||||
|
||||
IP-bound tokens prevent token theft across network boundaries. Do not remove the IP validation.
|
||||
|
||||
Token secret is loaded from `config/auth.yaml` at startup. It must never appear in source code or logs.
|
||||
|
||||
## Auth Backends
|
||||
|
||||
Backend selection is controlled by `config/auth.yaml`'s `provider:` field, matched literally against a `switch` in `backend/auth/init.go` — an unrecognized value (e.g. `configfile` without the underscore) fails apiserver startup with "unsupported auth provider". Valid identifiers: `config_file`, `pam`, `ldap`, `oauth2`. Only `config_file` (dev mode, users defined in the YAML) and `pam` (Linux + CGO only) are functional today — `ldap` and `oauth2` are unconditional stubs whose `Authenticate()` always returns "not yet implemented". All backends implement the same interface — adding a new backend means implementing that interface, not modifying middleware.
|
||||
|
||||
## Input Validation
|
||||
|
||||
Validate at the API boundary inside `preCheck{Resource}Request()` helpers — before any service or store access. Return 400 with a clear message on validation failure. Do not pass unvalidated input to downstream layers.
|
||||
|
||||
## Secrets
|
||||
|
||||
- JWT secret key: `config/auth.yaml` — should be file-system protected (e.g. chmod 0600); this is not currently enforced by the code or deploy tooling, so treat it as an operator responsibility, not a guarantee
|
||||
- LDAP bind credentials: `config/auth.yaml`
|
||||
- **Never log** tokens, passwords, secret keys, or full request bodies containing credentials
|
||||
- Validate all required config at startup; fail fast with a clear error if missing
|
||||
|
||||
## Rate Limiting and IP Whitelist
|
||||
|
||||
Rate limiting (`middleware/rate_limit.go`) and IP whitelist (`middleware/whitelist.go`, which implements both `WhitelistURLMiddleware()` and `WhitelistIPMiddleware()`) are applied before JWT validation in the middleware chain. They use `ctx.ClientIP()` and abort with 429 / 403 respectively. Both are configured in `config/` YAML — do not hardcode thresholds.
|
||||
|
||||
## Transport
|
||||
|
||||
All daemon-to-daemon communication is over TCP. The WAL and replication wire format uses Cap'n Proto — schema files in `src/schema/`. Do not add unvalidated deserialization paths for external input.
|
||||
|
||||
XCE uses **server-side TLS only** — encryption without peer certificate verification. There is no mTLS and no CA-pinning anywhere in the system; peer verification is deferred as optional P3 hardening (see bug-1453, which deleted all CA-cert/client-cert plumbing from both the C tier and the Go tier as dead code). Do not reintroduce CA-cert loading, client certificates, or cert-pinning without a design doc — this posture is intentional, not an oversight.
|
||||
|
||||
**Clients talking to the API server** (xcecli, or any HTTP client of `apiserver`) must support **both `http://` and `https://`** — the scheme is a per-deployment/user choice (`APIServerInfo.URL`), not something the client hardcodes. When the scheme is `https://`, set `InsecureSkipVerify: true` on the TLS client config (see `xcecli/common/request/request.go`'s `NewAPIClient`) rather than attempting cert validation — the API server's cert is typically self-signed/auto-generated and the client has no CA to validate it against.
|
||||
|
||||
**Clients talking to an XCE controller directly** (worker, scheduler, and any other libxce-based client) need **only server-side certs — no client-side certs are required or supported**. The live connection path is `xcecd_join_cluster()` (`src/worker/src/controller_connector.c`) calling `xcecli_start`/`xcecli_join` (`src/lib/client.c`), which drives `endpoint_connect()` in `src/lib/netio.c` over an `SSL_CTX` built by `create_ssl_context(NULL, NULL)` (`src/lib/xcectx.c`) — note that `cluster.c`'s `do_connect()`/`xce_connect_cluster()`/`xce_reconnect_init`/`xce_reconnect_step` are an unused stub path (explicitly documented as "to be wired in a later task") with no callers outside `cluster.c` and its own testsuite; don't point at them as the live mechanism. `xce_cluster_config_t` has no `ca_cert_path`/client-cert fields; do not add any. TLS verification mode is `SSL_VERIFY_NONE` throughout — this matches the API-server posture above, just at the C/libxce layer instead of Go/HTTP.
|
||||
|
||||
**Known follow-up (not yet scoped):** a HAProxy reverse proxy in front of `apiserver` is planned to terminate HTTPS, likely with a CA-trusted cert (Let's Encrypt candidate, unconfirmed), and whether `apiserver`'s own `RunTLS`/`httpsConfig.Enabled` listener stays in the picture is also undecided. Once both are settled, revisit xcecli's blanket `InsecureSkipVerify: true`: if the client-facing endpoint ends up CA-trusted, default to real verification (Go's default `tls.Config{}` validates against the system trust store with no extra plumbing) and make skip-verify an explicit opt-in for self-signed deployments only — matching the automated security review's suggested fix. Don't implement this speculatively; the right shape depends on which of those two decisions lands where.
|
||||
|
|
@ -1,106 +0,0 @@
|
|||
# Project Structure
|
||||
|
||||
## Organization Philosophy
|
||||
|
||||
**Layered monorepo, component-per-directory.** Each major subsystem lives under `src/<component>/` with its own Meson build file and CLAUDE.md. The Go frontend is isolated in `src/frontend/` as two independent Go modules. All C daemons share `src/lib/` (libxce) and public headers from `src/include/`.
|
||||
|
||||
```
|
||||
src/
|
||||
├── lib/ C — shared library (libxce): data structures, IPC, utilities
|
||||
│ ├── raft/ C — reusable Raft consensus module (raft.c/raft.h)
|
||||
│ └── testsuite/ C — libxce unit tests (test_*.c)
|
||||
├── controller/ C — cluster controller daemon
|
||||
├── worker/ C — per-node job execution daemon (xcecd)
|
||||
├── scheduler/ C — job scheduling engine (plugin architecture)
|
||||
├── autoscaler/ C — autoscaling engine (provider plugins)
|
||||
├── frontend/ Go — apiserver (Gin REST) + xcecli (Cobra CLI)
|
||||
├── xceadmin/ Python — admin CLI tool
|
||||
├── include/ C — public shared headers consumed across all daemons
|
||||
│ └── xce/ C — subset of public headers under the xce/ prefix
|
||||
├── schema/ Cap'n Proto schema files (c.capnp, controller.capnp)
|
||||
└── docs/ Project-wide documentation (markdown)
|
||||
```
|
||||
|
||||
## C Daemon Layout Pattern
|
||||
|
||||
`worker/` and `scheduler/` follow:
|
||||
|
||||
```
|
||||
src/<component>/
|
||||
meson.build — build definition
|
||||
CLAUDE.md — component-specific AI guide
|
||||
include/ — component-internal headers
|
||||
src/ — .c implementation files (flat, one domain per file)
|
||||
testsuite/ — test programs (test_*.c)
|
||||
```
|
||||
|
||||
**Exceptions**:
|
||||
- `controller/` keeps its `.c` files flat in the component root (not in a nested `src/`), and does have its own `testsuite/`.
|
||||
- `worker/` has no `testsuite/` — no `test_*.c` files exist anywhere in the component.
|
||||
- `autoscaler/` does not follow this pattern at all: it has no CLAUDE.md, and its daemon code lives two levels deeper in `src/autoscaler/scaler/` (flat `.c` files plus its own `include/` and `testsuite/`), alongside `src/autoscaler/providers/`.
|
||||
|
||||
Plugin-based components add their own subdirectory:
|
||||
- `scheduler/` — plugins in `src/plugins/`
|
||||
- `autoscaler/` — providers in `src/autoscaler/providers/`
|
||||
|
||||
## Go Frontend Layout Pattern
|
||||
|
||||
**apiserver** (`src/frontend/apiserver/`):
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| `cmd/` | Entry point, init sequence |
|
||||
| `router/` | Route registration + middleware chain |
|
||||
| `controllers/` | HTTP handlers — one file per resource |
|
||||
| `services/` | Business logic |
|
||||
| `store/` | Thread-safe in-memory KV stores (generics) |
|
||||
| `models/` | Request/response DTOs, domain structs |
|
||||
| `middleware/` | CORS, rate limiting, JWT, IP whitelist |
|
||||
| `backend/` | CGO bridge to C library |
|
||||
| `backend/auth/` | Pluggable auth providers |
|
||||
| `utils/` | Zap logger wrapper, config parsing, crypto |
|
||||
| `common/` | Global error codes and response helpers |
|
||||
| `config/` | YAML configs + JSON schemas |
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
### C
|
||||
|
||||
- **Files**: `snake_case.c` / `snake_case.h` — one domain per file
|
||||
- **Public headers**: `src/include/<name>.h` (e.g. `persist.h`, `netio.h`, `resreq.h`) — the libxce API boundary; a small subset (`client_failover.h`, `cluster.h`, `cluster_def.h`, `errnox.h`, `types.h`) lives under `src/include/xce/` and is included as `xce/<name>.h`
|
||||
- **Component headers**: `src/<component>/include/<name>.h`
|
||||
- **Test files**: `test_<unit>.c` (e.g., `test_ha_cluster.c`, `test_persist_wal.c`)
|
||||
- **Functions**: `snake_case` with component prefix — `ha_*`, `cluster_*`, `xcecd_*`, `xce_*`
|
||||
- **Types**: `snake_case_t` suffix (e.g., `cluster_t`, `ha_state_t`)
|
||||
- **Struct tags**: `struct <name>_s` (e.g., `struct cluster_s`)
|
||||
- **Enums/Macros**: `UPPER_SNAKE_CASE` with component prefix (`CTRL_PRIMARY`, `HA_DEFAULT_*`)
|
||||
|
||||
### Go
|
||||
|
||||
- **Files**: one file per resource — `jobs.go`, `queues.go`, `hosts.go`
|
||||
- **Tests**: `*_test.go` co-located; integration tests as `integration_test.go`, `<resource>_control_test.go`
|
||||
- **Controller structs**: `{Resource}Controller` (e.g., `JobController`, `ClusterController`, `QueueController`); one exception — `hosts.go` uses `HostHandler`
|
||||
- **Handler methods**: `{Action}{Resource}` (e.g., `CreateJob`, `GetJobsInfo`, `ControlJobByID`)
|
||||
- **Permission strings**: `"action:scope"` (e.g., `"job:view:self"`)
|
||||
- **JSON tags**: `snake_case`
|
||||
|
||||
## Test Organization
|
||||
|
||||
**C tests** (`testsuite/` inside each component):
|
||||
- Standalone `.c` programs, one per unit: `test_<unit>.c`
|
||||
- `test.c` provides the test runner harness
|
||||
- Tiered naming: `test_p1_integration.c`, `test_e2e_wal.c`, `test_p3_hardening.c`
|
||||
- Test-only symbols guarded by `#ifdef XCE_TESTING`
|
||||
|
||||
**Go tests** (co-located `_test.go`):
|
||||
- Table-driven unit tests, `t.TempDir()` for fixtures
|
||||
- Global state reset helpers (e.g., `resetRBACState()`) for test isolation
|
||||
- `make test` = unit only; `make test-all` = unit + integration
|
||||
|
||||
## Code Organization Principles
|
||||
|
||||
- **One domain per file** in C — `ha.c` owns HA logic, `job.c` owns job logic. No megafiles.
|
||||
- **Single Go module per binary** — apiserver and xcecli share no Go code at the module level.
|
||||
- **Controllers delegate to services** — HTTP handlers validate input and delegate; business logic lives in `services/`.
|
||||
- **Dependencies flow inward**: `controllers → services → store/backend`, never the reverse.
|
||||
- **Cap'n Proto for wire formats** — any new inter-daemon or persistence protocol uses `.capnp` schemas in `src/schema/`.
|
||||
|
|
@ -1,102 +0,0 @@
|
|||
# Technology Stack
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
xcecli (Go/Cobra) ──REST──► apiserver (Go/Gin) ──CGO──► C daemon layer
|
||||
controller / scheduler / worker / autoscaler
|
||||
↑↑↑ all linked against libxce (src/lib/)
|
||||
```
|
||||
|
||||
Request path inside apiserver: `Gin Router → Middleware (WhitelistURL → WhitelistIP → RateLimit → JWT → CORS) → RBAC Check → Controller → Services → In-Memory Store`
|
||||
|
||||
All controller state is in-memory. No persistent database in the API tier; job/queue/host state is pushed live into the in-memory stores from the C controller backend over CGO (see Key Technical Decision #2), not loaded from a file. C daemons communicate via TCP using a custom messaging layer: iomgr (epoll-based I/O loop), vtaskmgr (thread pool for packet dispatch), and xce_request/xce_reply for non-blocking send/receive. WAL + checkpoint provide persistence at the controller level.
|
||||
|
||||
## Core Technologies
|
||||
|
||||
- **Go 1.23.4** — apiserver (REST API) and xcecli (CLI). Two independent Go modules, no `go.work`.
|
||||
- **C (gnu17)** — controller, worker, scheduler, autoscaler, libxce. No C++.
|
||||
- **Gin v1.10.0** — HTTP router, middleware, binding
|
||||
- **Cobra v1.9.1** — CLI framework for xcecli
|
||||
- **Zap v1.27.0** — structured logging throughout (both Go modules)
|
||||
- **Viper v1.19.0 / v1.20.1** — YAML config management
|
||||
- **Cap'n Proto** — wire serialization for WAL and daemon-to-daemon replication
|
||||
- **iomgr / vtaskmgr / delaymgr** — libxce async I/O framework: single-threaded epoll loop (iomgr), thread-pool packet dispatch (vtaskmgr), timer thread for periodic/blocking tasks (delaymgr)
|
||||
|
||||
## Build Systems
|
||||
|
||||
| System | Scope |
|
||||
|--------|-------|
|
||||
| **Meson ≥ 1.5** | C daemons + Go modules via `tools/go_build` wrapper |
|
||||
| **Make** | Go development loop (build/test) from `src/frontend/` |
|
||||
| **Nix flakes** | Hermetic dev environment, AppImage packaging |
|
||||
|
||||
```bash
|
||||
# Full build (C + Go, debug + ASAN)
|
||||
/xce-build
|
||||
|
||||
# Go development loop (from src/frontend/)
|
||||
make build-server # → /tmp/apiserver
|
||||
make build-cli # → /tmp/xce_static (CGO_ENABLED=0)
|
||||
make test # unit tests
|
||||
make test-integration # regression suite (needs XCE_BIN + SRV_URL)
|
||||
|
||||
# Nix package builds (from src/)
|
||||
nix build .#xcecore # C daemons + apiserver
|
||||
nix build .#xcecore_debug # debug + ASAN
|
||||
nix build .#xcecli # CLI only
|
||||
```
|
||||
|
||||
## Development Standards
|
||||
|
||||
### Controller Pattern (Go)
|
||||
|
||||
```go
|
||||
// Struct: {Resource}Controller (typically an empty struct{})
|
||||
type JobController struct{}
|
||||
|
||||
// Handler: {Action}{Resource}
|
||||
func (j JobController) CreateJob(ctx *gin.Context) {
|
||||
// Validate input
|
||||
var req models.JobSubmitRequest
|
||||
if err := utils.BindJSONStrict(ctx, &req); err != nil { ReturnError(ctx, code, err.Error(), nil); return }
|
||||
parsedIndices, err := preCheckJobRequest(&req)
|
||||
if err != nil { ReturnError(ctx, code, err.Error(), nil); return }
|
||||
|
||||
// Business logic — package-level services function, not a controller field
|
||||
err = services.SubmitJob(&newJob, arraySize)
|
||||
if err != nil { ReturnError(ctx, code, err.Error(), nil); return }
|
||||
|
||||
ReturnSuccess(ctx, code, "submitted", result)
|
||||
}
|
||||
```
|
||||
|
||||
Early return on every error. Log with zap fields before returning.
|
||||
|
||||
### Testing Requirements
|
||||
|
||||
- **TDD mandatory**: Write failing test first, then implementation.
|
||||
- **Go unit tests**: Table-driven, no external dependencies, `t.TempDir()` for fixtures.
|
||||
- **C tests**: Add test case in `testsuite/` before modifying implementation.
|
||||
- `make test` must pass without a live server.
|
||||
- `make test-integration` requires `XCE_BIN` + `SRV_URL` env vars.
|
||||
|
||||
### Code Formatting
|
||||
|
||||
- Go: `gofmt` / `goimports` (post-edit hooks configured)
|
||||
- C: `clang-format` using project config at `xce/.clang-format`
|
||||
|
||||
### Auth and RBAC
|
||||
|
||||
- Auth backend set in `config/auth.yaml` (`provider: config_file | pam | ldap | oauth2`). Only `config_file` and `pam` (Linux + CGO) are functional; `ldap` and `oauth2` are documented as future/stub providers whose `Authenticate()` always returns a "not yet implemented" error.
|
||||
- Permissions are a plain action string (e.g. `"job:view"`, `"queue:submit"`) paired with a separate structured `domain` list (e.g. `{type: self}`, `{type: all}`) in `rbac.yaml` — not a combined `"action:scope"` string. `controllers.RequirePermission()` checks the action string only; domain/scope is evaluated separately.
|
||||
- RBAC loaded from `config/rbac.yaml`, enforced by `controllers.RequirePermission()` middleware.
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
1. **No Go workspace** — `apiserver` and `xcecli` are independent modules; changes to shared types require updating both.
|
||||
2. **In-memory API state** — no database in Go tier; `store.InitStores()` allocates empty per-resource stores at startup, and job/queue/host/etc. state is then pushed in live from the C controller backend over CGO (`backend.RegisterToBackend()` + `backend.StartNotificationListener()` in `cmd/main.go`). Only RBAC (`config/rbac.yaml`) and auth (`config/auth.yaml`) are actually loaded from YAML at startup.
|
||||
3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell.
|
||||
4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin.
|
||||
5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output.
|
||||
6. **Single-writer HA with full Raft log replication** — controller is always one active primary; standby election and log replication use a Raft protocol (term-based voting, freshness gates, `AppendEntries` request/reply, per-peer `next_index[]`/`match_index[]` progress vectors, `(term,index)` log matching/truncation, commit-index advance) implemented in the `src/lib/raft/` shared module (`raft.c`/`raft.h`) and validated by the `raft-log-replication`, `raft-commit-safety`, `raft-ha-core-safety`, and `ha-term-safety` specs.
|
||||
|
|
@ -1,95 +0,0 @@
|
|||
# Testing Standards
|
||||
|
||||
## Philosophy
|
||||
|
||||
- **TDD mandatory**: Write a failing test first, then the minimal implementation. The cycle is Red → Green → Refactor.
|
||||
- Test behavior, not implementation details.
|
||||
- Tests must run without a live server or external services.
|
||||
- A change is not complete until `make test` (and `make test-integration` where applicable) passes.
|
||||
|
||||
## Go Tests (apiserver / xcecli)
|
||||
|
||||
**Co-located** alongside source: `jobs.go` → `jobs_test.go`. Integration tests use descriptive names: `integration_test.go`, `host_control_test.go`.
|
||||
|
||||
**Table-driven** for validation/filter logic:
|
||||
|
||||
```go
|
||||
tests := []struct {
|
||||
name string
|
||||
input SomeRequest
|
||||
wantErr bool
|
||||
}{
|
||||
{"valid request", validReq, false},
|
||||
{"missing field", missingReq, true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := preCheckRequest(tt.input)
|
||||
if (err != nil) != tt.wantErr { t.Errorf(...) }
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
**State isolation** — reset global store state between tests:
|
||||
|
||||
```go
|
||||
func resetRBACState() { /* clear in-memory stores */ }
|
||||
|
||||
func TestPermissions(t *testing.T) {
|
||||
t.Cleanup(resetRBACState)
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
**Fixtures**: use `t.TempDir()` for file-system state; never rely on absolute paths or external services.
|
||||
|
||||
**Build targets**:
|
||||
- `make test` — unit tests only (`CGO_ENABLED=0`)
|
||||
- `make test-backend` — CGO-backed backend tests (requires nix dev shell + build dir)
|
||||
- `make test-integration` — full regression suite (requires `XCE_BIN` and `SRV_URL` env vars)
|
||||
|
||||
## C Tests (controller / scheduler / lib)
|
||||
|
||||
**Location**: `testsuite/` directory inside each C component.
|
||||
|
||||
**Note**: `src/worker` has no `testsuite/` directory and no meson `test()` registrations today — it currently carries no C test coverage of its own.
|
||||
|
||||
**Structure**: one `.c` test program per unit:
|
||||
|
||||
```
|
||||
src/controller/testsuite/
|
||||
test.c ← test runner harness
|
||||
test_ha_cluster.c ← tests for ha.c
|
||||
test_persist_wal.c ← WAL-specific tests
|
||||
test_e2e_checkpoint.c ← end-to-end checkpoint flow
|
||||
test_p1_integration.c ← P1 HA integration tier
|
||||
```
|
||||
|
||||
File naming convention: `test_<unit>.c`. E2E tests: `test_e2e_<topic>.c`. Integration tiers: `test_p1_*.c`, `test_p2_*.c`, `test_p3_hardening.c`.
|
||||
|
||||
**Test-only symbols** are guarded to prevent production bloat:
|
||||
|
||||
```c
|
||||
#ifdef XCE_TESTING
|
||||
extern void reset_ha_state(ha_state_t *s);
|
||||
#endif
|
||||
```
|
||||
|
||||
**Add a test case in `testsuite/` before modifying any C implementation.**
|
||||
|
||||
## Verifying a Test Run Actually Passed
|
||||
|
||||
**Meson's aggregate `Ok:N Fail:0` is not sufficient evidence.** Meson can report a suite `Fail:0` while Unity sub-groups inside it fail — meson only tracks the process exit code of each *meson test unit*, not each Unity `RUN_TEST`. Always cross-check the **standalone test binary's own stdout** (`controller_unit_test`, `xcelib_test`, etc., run directly) for its authoritative `:FAIL:` count / `N Tests M Failures` summary before treating a run as green. This was the root cause of a masked-failure bug in the Unity harness itself (nested `UNITY_BEGIN()`/`UNITY_END()` across 83 `controller/testsuite/*.c` files swallowed the real exit code — fixed 2026-07-17) and is now the standard verification step for every `/kiro-validate-impl` pass.
|
||||
|
||||
## Coverage Targets
|
||||
|
||||
- Go: 80% minimum on business logic (controllers, services, store)
|
||||
- C: every new function in a daemon must have at least one test case in `testsuite/`
|
||||
- Integration: all public REST endpoints must have at least one integration test case in `src/frontend/tests/regression/`
|
||||
|
||||
## What Not to Mock
|
||||
|
||||
- Mock **external dependencies only** (network calls, file I/O that crosses component boundaries)
|
||||
- Do **not** mock the system under test
|
||||
- In-memory stores (`store/`) are real — use them directly in unit tests
|
||||
- CGO bridge (`backend/`) is covered by `make test-backend` — do not stub it in unit tests
|
||||
Loading…
Add table
Add a link
Reference in a new issue