lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

update documents

Browse source
This commit is contained in:
Rongsong Shen 2026-08-10 17:51:51 +08:00
parent c705cadadc
commit d70bf6a402
51 changed files with 995 additions and 1997 deletions
Download patch file Download diff file Expand all files Collapse all files

37
src/controller/CLAUDE.md
View file

@ -13,9 +13,18 @@ For full build system details, see `.kiro/steering/build-deployment.md`.
Core source files handle distinct domains:
- `job.c` — Job lifecycle management
- `queue.c` — Queue state and control
- `cluster.c` — Cluster topology
- `cluster.c` — Cluster topology, request dispatch
- `dataex.c` — Data exchange with other daemons
HA / replication / persistence subsystem (see HA Protocol section below for the election/replication/commit design; functions from these files are what the TLA+ mapping and safety invariants below actually cite):
- `ha.c` — HA state machine, election glue, promotion/fencing logic
- `ha_convergence.c` — post-election convergence and recovery
- `persist.c` — WAL persistence, durability, recovery
- `repl.c` — replication broadcast, majority-ack, WAL streaming
- `config_repl.c` — configuration replication and catch-up
`src/controller/` has ~27 `.c` files total (`ls src/controller/*.c`); the remainder (`commit.c`, `ctrlstatus.c`, `join.c`, `worker.c`, `resources.c`, `task.c`, and others) cover commit-index advancement, status reporting, node join, and worker/resource/task bookkeeping.
Links against `libxce` from `src/lib/`.
## Conventions
@ -45,12 +54,12 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
| TLA+ Variable | XCE Code | Location |
|--------------|----------|----------|
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:139` |
| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:139` |
| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:161` |
| `currentTerm[i]` | `ha->term` (_Atomic uint64_t) | `ha.h:157` |
| `state[i]` ∈ {Follower,Candidate,Leader} | `ha->role` ∈ {INIT,ELECTING,PRIMARY,STANDBY} | `ha.h:158` |
| `votedFor[i]` | `ha->voted_for_name[XCE_MAXNAME_LEN]` | `ha.h:179` |
| `log[i]` | WAL segments in `persist->wal_path` | `persist.c` |
| `commitIndex[i]` | `persist->committed_lsn` (_Atomic uint64_t) | `persist.h` |
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | `ha.h:141` |
| `lastApplied[i]` | `ha->applied_lsn` (_Atomic uint64_t) | `ha.h:160` |
| `nextIndex[i,j]` | `standby.join_applied_lsn` (replicator) | `repl.h` |
| `matchIndex[i,j]` | `standby.acked_lsn` (replicator) | `repl.h` |
@ -58,23 +67,23 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
| TLA+ Action | XCE Code | Location |
|------------|----------|----------|
| `Timeout(i)` | `raft_start_election` via delaymgr | `raft.c:57` |
| `RequestVote(i,j)` | `raft_on_vote_request` / reply path | `raft.c:106,169` |
| `BecomeLeader(i)` | `raft_cb_become_leader` | `ha.c:347` |
| `AdvanceCommitIndex(i)` | `repl_wait_majority_ack` → advance `committed_lsn` | `repl.c:338` |
| `Timeout(i)` | `raft_start_election` via delaymgr | `raft.c:105` |
| `RequestVote(i,j)` | `raft_on_vote_request` / reply path | `raft.c:178,433` |
| `BecomeLeader(i)` | `raft_cb_become_leader` | `ha.c:1615` |
| `AdvanceCommitIndex(i)` | `repl_wait_majority_ack` → advance `committed_lsn` | `repl.c:589` |
| `AppendEntries(i,j)` — production send (LIVE) | `repl_broadcast` (per-entry fan-out on append, NACK-driven retry via `ha_raft_on_repl_nack`) + `ha_stream_wal_to_standby` (catch-up-range streaming) — the sole production `AppendEntries`-equivalent send/retry/backoff mechanism (raft-adapter-conformance Req 3.3) | `repl.c:295`, `config_repl.c:2351` |
| `AppendEntries(i,j)` — nextIndex-driven sender (DORMANT, documented) | `raft_send_append_entries` is fully protocol-conformant but has ZERO production callers (only `raft_sim.c` and an `XCE_TESTING` test helper); its registered `on_send_append_entries` callback `ha_cb_send_append_entries_stub` is reachable only through it. Kept as an intentionally-dormant reference implementation (retire/wire-in both rejected, see `research.md`), guarded against silent regression by `ha_send_append_entries_stub_call_count` staying 0 while the production path fires (`test_ha_core_safety.c` task 4.2) | `src/lib/raft/raft.c:595`, `ha.c:2590` (stub), `ha.c:2588` (guard counter) |
| `AppendEntries(i,j)` reply — WAL catch-up reject | Every WAL catch-up rejection sends an explicit `WalAckReply{success=0}` via `ha_send_wal_ack_reply_ex` back to the primary — never a silent drop (bug-1479; reply-construction boundary regression-locked by raft-adapter-conformance task 5.1) | `ha.c:3155` (fn), `ha.c:3190` (accept call site), `ha.c:3326` (reject call site) |
| `AppendEntries(i,j)` reply — configuration-push accept/reject | Every `handle_repl_config_push` outcome — all 4 reject branches plus the success/`-EAGAIN`-activation-deferred tail — sends exactly one explicit `ReplConfigPushReply{accepted, term}` via `send_config_push_reply`, so the primary can distinguish an active reject from a lost push (raft-adapter-conformance Req 2.2-2.5); the reply's `term` field feeds the primary's `raft_on_higher_term_observed` term-adoption alongside the heartbeat and WAL-NACK vectors (Req 7.4/7.5) | `config_repl.c:1267` (send fn), `config_repl.c:1308-1381` (handler), `cluster.c:750-780` (primary-side accept/reject branch + term-adoption) |
| `AppendEntries(i,j)` reply — configuration-push accept/reject | Every `handle_repl_config_push` outcome — all 4 reject branches plus the success/`-EAGAIN`-activation-deferred tail — sends exactly one explicit `ReplConfigPushReply{accepted, term}` via `send_config_push_reply`, so the primary can distinguish an active reject from a lost push (raft-adapter-conformance Req 2.2-2.5); the reply's `term` field feeds the primary's `raft_on_higher_term_observed` term-adoption alongside the heartbeat and WAL-NACK vectors (Req 7.4/7.5) | `config_repl.c:1267` (send fn), `config_repl.c:1308-1381` (handler), `cluster.c:1030-1055` (primary-side accept/reject branch + term-adoption) |
| `RequestVote`/`PreVote(i,j)` reply — endpoint-failure routing, all timings | A vote/PreVote endpoint failure is always counted through the production tally (`vote_reply_dispatch` / `pre_vote_reply_dispatch`), never the legacy `handle_request_vote_reply` path — true both at connect-time (bug-1423/1424, regression-locked) and post-connect, i.e. a failure after connect but before a reply arrives (raft-adapter-conformance Req 4.2/4.3, closing that one remaining failure-timing gap). `vote_ep_err_handler` (real-vote) and `pre_vote_ep_err_handler` (PreVote) are separate handlers sharing one `CTRL_PRIMARY`-cleanup helper | `ha.c:7667` (`vote_ep_err_handler`), `ha.c:7705` (`pre_vote_ep_err_handler`), `ha.c:7627` (shared cleanup helper), registered at `ha.c:692` / `ha.c:831` |
| `ClientRequest(i,v)` | Job submit → WAL append → `repl_wait_majority_ack` | `persist.c`, `repl.c` |
| `UpdateTerm(i,j,m)` | Term CAS in `handle_request_vote` / vote reply handler | `ha.c:1218-1230` |
| `UpdateTerm(i,j,m)` | Term CAS before granting/casting a vote — now lives in the Raft library, not the adapter | `raft.c:178-270` (`raft_on_vote_request`) via `raft_cb_persist`, `ha.c:2257` |
> Full gap-analysis citations, options considered, and design rationale for the `raft-adapter-conformance` rows above (including the dormant-sender decision and the vote/PreVote handler-split decision): `.kiro/specs/raft-adapter-conformance/research.md` and `design.md`. This table stays the terse, current pointer; the evidentiary trail lives there.
### Safety Invariants Verified
**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). A candidate needs majority votes; each server votes at most once per term.
**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request`'s duplicate-vote check, `raft.c:231-237`). A candidate needs majority votes; each server votes at most once per term.
**Leader Completeness** (committed entries survive): Freshness gate in `RequestVote` — candidate must have `lastLogTerm ≥ voter` AND `lastLogIndex ≥ voter`. Since committed entries are on a majority, the new leader must have all committed entries. The post-election truncation removes only a divergent tail **above the leader's own durable log**: the cutoff is `ha_promotion_truncate_floor(safe_committed_lsn, durable_tip) = max(safe_committed_lsn, persist_durable_floor)`, so `persist_truncate_wal_above` is a no-op on the leader's own durable entries. **A new leader MUST NEVER truncate its own log** — entries it holds durably were acked by it (as a standby) and so were committed by a prior leader; truncating to the bare majority element destroys them (bug-1060: granted `{self=47, peer=44}` → `safe=44` cut the leader's own committed 45..47 → divergent WALs + permanent prevote deadlock).
@ -101,7 +110,7 @@ The `RequestVoteReply` message already carries `peerConfigTag` and `peerLastLogL
### Protocol Verification Checklist
When modifying HA protocol code (election, replication, heartbeat), verify:
- [ ] `votedFor` is updated atomically with term increment before granting any vote (library: `raft_on_vote_request`, `src/lib/raft/raft.c:178-270` — term update before persist, `voted_for` set only after persist succeeds; adapter: `raft_cb_persist`, `ha.c:1784-1824`, writes term+voted_for atomically under `ha->lock`. `handle_request_vote` in `ha.c` is `#ifdef XCE_TESTING`-only; production dispatch is `cluster.c:477` → `raft_on_vote_request`. Corrected 2026-07-18 — the prior `ha.c:1218-1230` citation had drifted onto unrelated post-election code.)
- [ ] `votedFor` is updated atomically with term increment before granting any vote (library: `raft_on_vote_request`, `src/lib/raft/raft.c:178-270` — term update before persist, `voted_for` set only after persist succeeds; adapter: `raft_cb_persist`, `ha.c:2257`, writes term+voted_for atomically under `ha->lock`. `handle_request_vote` in `ha.c` is `#ifdef XCE_TESTING`-only; production dispatch is `cluster.c:582` → `raft_on_vote_request`. Re-corrected 2026-08-08 — the `ha.c:1784-1824` and `cluster.c:477` citations from the prior 2026-07-18 correction had themselves drifted onto unrelated promotion/recovery-gate code.)
- [ ] `currentTerm` is monotonic — never decreases
- [ ] `commitIndex` never decreases (monotonic)
- [ ] A leader only commits entries from its current term (or equivalent via per-entry majority ack)
@ -121,8 +130,6 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
Covered in `.kiro/steering/ha-async-io.md` (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and `.kiro/steering/build-deployment.md` (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference.
## Key Files
## See Also
- Shared library: `src/lib/CLAUDE.md`
- Root guidelines: `CLAUDE.md` (repo root)

2
src/docs/FUTURE_WORK.md
View file

@ -3,5 +3,5 @@
> **Moved (2026-06-10)**: All items merged into the single consolidated future-work document at
> [`.kiro/steering/roadmap.md`](../../.kiro/steering/roadmap.md).
>
> Section mapping: §1+§8 → admin-config-tool (backlog); §2 → superseded by ha-consistency-invariants Req 5;
> Section mapping: §1+§8 → admin-config-tool (nearly complete — see `.kiro/specs/admin-config-tool/`); §2 → superseded by ha-consistency-invariants Req 5;
> §3, §4, §5, §6, §7 → Hardening / Direct-Implementation Items.

10
src/docs/apiserver-backend-tasks.md
View file

@ -11,7 +11,7 @@
- [x] **#1 P0-1a: Add HostConfig + Cluster fields to models**
- `models/clusters.go` — Added `HeartbeatInterval`, `Vcpus`, `InactiveHeartbeatCount` to `ClusterInfo`
- `models/hosts.go` — Changed `HostStateStr(0)` from `"OK"` to `"UNKNOWN"`
- `models/hosts.go` — This task changed `HostStateStr(0)` from `"OK"` to `"UNKNOWN"`; commit `d7afd72e` ("fix front end issues") later reverted that branch, so `HostStateStr(0)` currently returns `"OK"` again
- `models/clusters_test.go` — New: verifies new fields serialize + existing fields unchanged
- `models/hosts_test.go` — New: verifies `HostStateStr` for zero, flags, non-flag, combined
- All 15 tests passing
@ -51,7 +51,7 @@
- `include/xceapi.h` — Added `xce_hostctrl()` declaration
- `include/request.h` — Added `xce_hostctrl_req()` declaration
- `lib/client.c` — Implemented `xcecli_hostctrl()`
- `lib/xceapi.c` — Implemented `xce_hostctrl()` with `cbuf_get()` wait
- `lib/xceapi.c` — Implemented `xce_hostctrl()` with `cbuf_get()` wait (a later, unrelated repo-wide change converted this to `cbuf_timedget()` with a 10s deadline)
- `lib/request.c` — Implemented `xce_hostctrl_req()` (validates HOST_OPEN/HOST_CLOSE only, rejects HOST_TAG/UNTAG)
- `lib/helper.c` — Added `Request_body_hostctrlmsg` case to `xce_request_body()` + `xce_request_msg_name()`
- `controller/cluster.c` — Added `OPCode_hostcontrol` case → `do_hostcontrol_op()`
@ -77,7 +77,7 @@
- All tests passing (`controllers` package)
- [x] **#9 P0-2d: Update stub.go with new function stubs**
- `backend/stub.go` — Added `StartHeartbeat(_ context.Context){}`, `SetHeartbeatInterval(_ int){}`, `EvalSelectForHost(_, _ string) error` (returns `errNoCGO`), `Cleanup(){}`, added `"context"` to imports
- `backend/stub.go` — Added `StartHeartbeat(_ context.Context){}`, `SetHeartbeatInterval(_ int){}`, `EvalSelectForHost(_, _ string) error` (returns `errNoCGO`), `Cleanup(){}`, added `"context"` to imports (`SetHeartbeatInterval` was later pulled out of `stub.go`/`clib.go` entirely into the shared, no-build-tag `backend/heartbeat.go`)
- `backend/stub_funcs_test.go` — 4 TDD tests: `TestStartHeartbeat_StubNoOp`, `TestSetHeartbeatInterval_StubNoOp` (3 edge values), `TestEvalSelectForHost_StubErrNoCGO`, `TestCleanup_StubNoOp`
- All backend tests passing
@ -106,7 +106,7 @@
## P1-3: Heartbeat Keep-Alive
- [x] **#13 P1-3: Heartbeat keep-alive implementation**
- `backend/clib.go` — Added `heartbeatIntervalSecs int32` (default 30), `SetHeartbeatInterval()`, `heartbeatFn` var, `StartHeartbeat(ctx context.Context)` goroutine with ticker
- `backend/clib.go` — Added `heartbeatIntervalSecs int32` (default 30), `SetHeartbeatInterval()`, `StartHeartbeat(ctx context.Context)` goroutine with ticker (within the same P1-3 commits, `heartbeatIntervalSecs`/`SetHeartbeatInterval()`/the tick loop were extracted into the shared, no-build-tag `backend/heartbeat.go` for CGO-free testability; `clib.go` now passes an inline closure to `runHeartbeat` rather than a named `heartbeatFn` var)
- `backend/handler.go` — `OnClusterUpdated` calls `SetHeartbeatInterval(cluster.HeartbeatInterval)` after store update (zero-preserve: skips if interval==0)
- `cmd/main.go` — Calls `backend.StartHeartbeat(ctx)` after `backend.RegisterToBackend(...)`
- `backend/handler_test.go` — Tests for heartbeat interval update, zero-preserve, timer leak, test drain, store consistency
@ -123,7 +123,7 @@
- Input safety — comment limit 512 chars; select expression limit 4094 chars (1024 reduction deferred to P1-2b)
- **CGO fix applied:** extracted `static` val helpers from `clib.go` preamble into `val_helpers.h` shared header; `convert.go` now includes it; added explicit `float64()` cast for `C.double` return
- [x] **#15 Integration test suite** — see [`apiserver-backend-integration-test-plan.md`](apiserver-backend-integration-test-plan.md)
- [x] **#15 Integration test suite** — plan doc `apiserver-backend-integration-test-plan.md` was later removed from `src/docs/` in an unrelated cleanup commit; see the test files below for current coverage
- `backend/integration_test.go` — §1 (config push), §3 (resource values), §4 (heartbeat), §5 (cross-feature): 17 tests
- `controllers/integration_test.go` — §2 (host control), §6 (RBAC), §8 (input safety): 17 tests + 1 skip (§8.1 pending P1-2b)
- `store/concurrency_test.go` — §7 (thread safety): 3 concurrent-access tests

2
src/docs/archive/cluster-config-split-analysis.md
View file

@ -1,3 +1,5 @@
> **ARCHIVED 2026-08-08.** This analysis's central subject — the ClusterConfig struct and its proposed 3-way Group1/PeersConfig/ControllerConfig split — no longer describes the schema. The closed `clusterconfig-schema-split` spec implemented a different, simpler design; `MasterCandidate`, `candidates`, and `ClusterConfig.port` (which this doc analyzes as fields to remove/split) are already gone from the schema. Kept for historical context only — do not treat any field/line reference below as current.
# ClusterConfig Schema Split Analysis
> Analysis date: 2026-06-08. Branch: persistent_ha.

2
src/docs/archive/job_peek.md
View file

@ -1,3 +1,5 @@
> **ARCHIVED 2026-08-08.** This implementation plan is stale throughout — the C-layer fetchfile pipeline it describes as "not started" is fully implemented, `output.go` already uses the "Plan B" (CGO/FetchFile) path this doc proposes migrating to, and opcode/struct layouts have drifted from what shipped. Note: the shipped `handle_fetchfile()` worker handler has no path-traversal or job-ownership validation — a real gap this doc does not flag. Kept for historical context only.
# 项目实施方案：xcecli jobs peek
## 背景

356
src/docs/config-replication-design.md
View file

@ -142,7 +142,8 @@ The full design is chosen for clusters where config integrity verification, chan
auditability, and the push mechanism for online config updates are operational requirements.
**WAL-based config replication (considered and rejected)**: XCE already has a WAL
replication path (`src/controller/repl.c`, ~340 lines) that streams job/queue/resource
replication path (`src/controller/repl.c`, 881 lines as of this audit — grown from the
~340 lines at design time) that streams job/queue/resource
mutations from primary to standbys. Extending it to carry serialised `ClusterConfig`
blobs would reuse the existing replication infrastructure. This approach is rejected
for three reasons: (1) **Semantic mismatch** — WAL replication is continuous and
@ -187,8 +188,12 @@ and `validate_host()` are complete; see `FUTURE_WORK.md`):
- ~170 LOC
The `OP_upgradeConfig` handler is gated behind a compile-time flag
`CONF_ENABLE_UPGRADE_CONFIG` (default: 0, disabled — see §5.3 safety gate). The
gate MUST NOT be set to 1 in any production build until BOTH prerequisites are met:
`CONF_ENABLE_UPGRADE_CONFIG`. **As designed**, the default is 0 (disabled — see
§5.3 safety gate) and the gate MUST NOT be set to 1 in any production build
until BOTH prerequisites are met. **As shipped**, `src/controller/config_repl.h`
hardcodes the flag to `1` (enabled) with no Meson build option to override it —
see the drift note in §5.3. Both prerequisites below remain unmet in the
current codebase:
1. **`validate_host()`** is implemented with a real authorisation check (not a stub
that returns success for all peers). Minimum viable check: verify the peer's
@ -206,6 +211,11 @@ active, Phase 2 infrastructure compiled but inert). The gate is removed from the
codebase when both prerequisites are merged and integration-tested. Phase 1 alone
eliminates manual provisioning and config drift — the core problems stated in §1.
**This commitment is not currently met**: the codebase already ships with
`CONF_ENABLE_UPGRADE_CONFIG` hardcoded to `1` while `validate_host()` remains
the unconditional-success stub — the exact unsafe combination this gate was
designed to prevent. See the drift note in §5.3.
---
## 3. Config Storage Layout
@ -221,7 +231,7 @@ would require a separate atomic-rename technique anyway.
**Local filesystem only — NFS is explicitly unsupported.** The controller's working
directory (including `conf/`, `data/wal/`, and `data/snap/`) MUST reside on a local
filesystem. `persist_init()` (persist.c:217-222) detects and rejects NFS mounts at
filesystem. `persist_init()` (persist.c:839-843) detects and rejects NFS mounts at
startup with a FATAL error. Config replication assumes local `rename(2)` atomicity
and `fsync(2)` durability guarantees; neither holds on NFS. No effort is made to
support network or distributed filesystems for config storage. Cross-filesystem
@ -615,6 +625,18 @@ half-implemented admin endpoint from being reachable in production before its
authorisation check is complete. The compile-time gate is removed once
`validate_host()` has a production implementation (tracked in `FUTURE_WORK.md`).
**Drift from shipped code**: The compile-time default described above was the
design intent and is not what the current codebase does. As implemented,
`src/controller/config_repl.h` defines `CONF_ENABLE_UPGRADE_CONFIG` to `1`
unconditionally (comment: "enabled: standbys apply pushed config"), and
`src/controller/meson.build` has no `-Denable-upgrade-config` option or any
override of the flag — the compile-time default cannot be reverted to 0
without editing the header. Meanwhile `validate_host()` (`src/controller/join.c`)
is still the literal stub `return 1;` for every peer. `OP_upgradeConfig` is
therefore reachable in any build of the current codebase without the
authorisation check this gate was meant to require. This should be treated as
an open safety gap, not a documentation nit.
### 5.4 Primary handler
The primary handles `OP_replConfig` via a new case in `cluster.c:myreq_handler()` (the
@ -1320,6 +1342,17 @@ load_primary_config() INFO("Starting as standby
cluster_start()
```
> **Implementation note — startup ordering changed**: The diagram above
> reflects this design's original proposal (election before
> `iomgr_init`/`endpoint_init`). The shipped implementation moves the listener
> and `iomgr_init`/`vtaskmgr_init` **before** election instead, closing a
> 2–3 second window during which the port was unreachable to election probes.
> See `.kiro/specs/ha-config-bootstrap/design.md`
> (`Net[iomgr_init + cluster_open_listener + vtaskmgr_init]` runs before
> `Quorum[am_i_primary_quorum]`) — this ordering change accompanies the
> replacement of `am_i_primary()` with Raft-style quorum election (§9
> archival note).
### 7.1 Primary config load — fail-fast behaviour
`load_primary_config()` replaces `get_default_conf()` on the primary path:
@ -1385,6 +1418,10 @@ From the current `cluster_init()` (`cluster.c`):
### 8.2 What stays in phase 2 (after config)
> **Implementation note**: As shipped, `iomgr_init`/`endpoint_init` (and
> `vtaskmgr_init`) run **before** election, not after — see the ordering note
> in §7. The table below reflects this design's original phase-2 placement.
| Line(s) | Code | Phase |
|---|---|---|
| new (before iomgr) | Stop heartbeat listener: set `atomic_store(&c->heartbeat_listener_running, 0)` and `pthread_join(c->heartbeat_listener_tid, NULL)` — the join must complete before `iomgr_init` to free the listening port and prevent `EADDRINUSE` | 2 |
@ -1460,288 +1497,26 @@ the primary's startup log and tune `max_attempts` × `max_delay_ms` accordingly.
## 9. am_i_primary() Implementation
The current `am_i_primary()` stub (`ha.c:213`) returns `c->ha.my_priority == 0`,
which is never correct for a multi-node cluster — it never probes whether a
higher-priority candidate is actually alive.
### 9.0 Simultaneous-boot symmetry breaking
When all candidates start simultaneously, nobody is primary yet and nobody responds
to election probes. Without symmetry breaking, every candidate would expire its
deadline and self-declare primary — a multi-way split-brain.
The mechanism has two parts:
**1. Dedicated heartbeat listener (all self-declared primaries)**
After Phase 1, **every candidate that `am_i_primary()` returns true for** launches a
minimal heartbeat listener thread before entering Phase 2. This is not limited to
priority-0 — if a lower-priority candidate wins the jitter race and self-declares
primary, its heartbeat listener is essential for the higher-priority true primary
to detect the split-brain during its own election probe. The listener is started
in a dedicated pthread immediately after `am_i_primary()` returns:
```c
if (am_i_primary()) {
/* Start a minimal SSL listener that responds to OP_heartbeat probes.
* This lets lower-priority candidates detect us while Phase 2 initializes. */
start_heartbeat_listener(c); /* dedicated pthread — non-blocking poll loop */
load_primary_config(c); /* FATAL if missing */
...
}
```
The heartbeat listener is fully specified in the parent HA design
(`persistence-and-ha-design.md` Part 5). This section only describes the
integration point relevant to config replication: the listener must be active
before the standby's `fetch_config_from_primary()` connects, so the standby can
reach a primary that is still in Phase 2 initialisation. Summary of the
listener's properties (see parent design for full specification):
- Binds the port from `--peers` using `c->ha.ssl_ctx`
- Non-blocking poll loop; accepts `OP_heartbeat` only; rejects other opcodes
- Does NOT depend on iomgr, vtaskmgr, persist, or any Phase 2 component
- Exits when the main thread clears `atomic_int heartbeat_listener_running`
(set to 0 before `iomgr_init` in Phase 2). The main thread calls
`pthread_join(heartbeat_listener_tid, NULL)` before `endpoint_init()` to
prevent EADDRINUSE.
- Registered in `cluster_t` (`c->heartbeat_listener_tid`,
`c->heartbeat_listener_running`) so `cluster_fini()` can join the thread
on any FATAL exit path before Phase 2
**2. Priority-based base delay + randomized jitter (all candidates)**
Each candidate's election timeout has two components: a deterministic
**priority-based base delay** (higher priority = shorter delay) plus a small
random jitter for tie-breaking:
```c
/* Priority-based base delay: priority-0 waits 0ms, priority-1 waits
* election_priority_step_ms, priority-2 waits 2× that, etc.
* This ensures the highest-priority node nearly always wins without probing. */
long to_ms = c->ha.config.election_timeout_ms;
to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms;
/* Small random jitter for tie-breaking between same-step candidates */
long jitter_ms = (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms);
to_ms += jitter_ms;
```
With `election_timeout_ms = 2000`, `election_priority_step_ms = 500`, and
`election_jitter_max_ms = 200`, the effective deadlines are:
- Priority-0: 2000ms–2199ms (self-declares primary first)
- Priority-1: 2500ms–2699ms (probes priority-0 first; if alive, becomes standby)
- Priority-2: 3000ms–3199ms (probes priority-0 and priority-1 first)
The priority-based base delay all but eliminates simultaneous split-brain during
normal boots: priority-0 declares primary and starts its heartbeat listener
~500ms before priority-1's deadline, so priority-1's probe finds a live primary.
The priority-based step and jitter together make simultaneous split-brain
vanishingly unlikely for N=3 in systemd-managed deployments:
- Priority-0 fires at t=0–199ms (election_timeout_ms + jitter only)
- Priority-1 fires at t=500–699ms (step + jitter, probes priority-0 first)
- Priority-2 fires at t=1000–1199ms (2×step + jitter)
Adjacent-priority candidates collide only if the lower-priority node starts so
late that its jitter-extended deadline precedes the higher-priority node's first
heartbeat response, which requires the 500ms step to be less than 2×J_max
(400ms) — false for the configured values. For three nodes to all self-declare
primary, all three must start within the same ~200ms jitter window, which is
bounded by OS scheduler precision and systemd startup ordering.
The `election_priority_step_ms` is added to `ha_config_t` alongside the existing
`election_timeout_ms` and `election_jitter_max_ms`. The jitter-only fallback
(priority-0 with no step) retains the original behaviour when `election_priority_step_ms = 0`.
The window is bounded: in the worst case (all jitter values 0ms apart), all
candidates declare primary. Once both reach Phase 2, the HA watchdog resolves
the split via priority comparison in heartbeat exchange (see parent HA design).
This is bounded by the time for the slower candidate to complete Phase 2
(~4–13s per §8.5) plus one heartbeat interval (~500ms).
**Split-brain window during simultaneous boot**: The heartbeat listener starts
immediately after `am_i_primary()` returns, but it is a minimal pre-iomgr
listener that only handles `OP_heartbeat`. **All candidates that self-declare
primary launch a heartbeat listener**, not just priority-0 — this is critical:
if a lower-priority candidate wins the jitter race and self-declares primary,
its heartbeat listener provides the response that resolves the split-brain
when the higher-priority (true) primary probes it.
During simultaneous boot, all candidates start with `ha.term = 0`
(`atomic_store_explicit(&ha->term, 0ULL, ...)` in `ha_init()`, ha.c:34).
Term-based priority comparison cannot break symmetry at this stage — symmetry
breaking during simultaneous boot relies entirely on the jitter mechanism
(part 2 below). Term-based step-down becomes effective only after the first
successful election, when the winning primary increments its term. The jitter
window is therefore the primary defence against multi-way split-brain during
cold start; the heartbeat listener provides the detection channel but term
comparison is not yet discriminating.
If a heartbeat listener fails to bind its port (`EADDRINUSE`), the candidate
logs a WARN indicating a likely split-brain condition (another candidate has
already bound the port) and continues initialization. This is expected during
simultaneous boot with overlapping jitter windows — the candidate that bound
the port first keeps it; the other continues to Phase 2 where the HA watchdog's
heartbeat exchange will resolve the split-brain.
If a lower-priority candidate also
self-declares primary and enters Phase 2, the HA watchdog heartbeat exchange
will resolve the split-brain once both candidates reach Phase 2 — but this
requires the HA watchdog to compare candidate priority (from `--peers` ordering)
in addition to term, since both nodes have term=0 during simultaneous boot.
**This design implements** `ha_check_heartbeat_nocluster()` (ha.c:101-126)
with a priority comparison at term=0 so that a lower-priority self-declared primary
steps down when it receives a heartbeat from a higher-priority node with the
same term. Without this enhancement, the current elapsed-time-only check cannot
distinguish a legitimate primary from a split-brain peer during cold start.
**CRITICAL DEPENDENCY — implemented by this design**: The `ha_check_heartbeat_nocluster()`
enhancement (priority comparison at term=0) is a **hard prerequisite** for correct
simultaneous-boot behavior. Without it, a lower-priority self-declared primary will
not step down when it receives a heartbeat from a higher-priority peer with the same
term — the split-brain is unbounded, not 4–13s. This enhancement is implemented in
this design (`ha.c` — see §12 files changed) alongside the election probing. The
parent HA design (`persistence-and-ha-design.md` Part 5) specifies the heartbeat
exchange requirements; this design provides the priority-comparison implementation
which the parent design references. Test coverage:
`test_am_i_primary_simultaneous_boot` in §14.1 validates priority-based step-down
at term=0 during simultaneous boot.
**Implementation scope**: This design adds a `sender_priority` field to the
`OP_heartbeat` request in `controller.capnp`. The heartbeat listener (§9.0)
populates this field from `c->ha.my_priority`. `ha_check_heartbeat_nocluster()`
(ha.c) compares `sender_priority` against `ha->my_priority` when `term == 0`:
if the sender has higher priority (lower index in `--peers`), the local node
steps down to standby. The data source is the heartbeat message, not `ha_state_t`,
so both the schema change and the handler change are part of this design.
The effective split-brain window is therefore the time for the
secondary-primary to complete Phase 2 and start processing heartbeats: ~4–13s
per §8.5. During this window both self-declared primaries have `iomgr_init` and
`endpoint_init` complete (per the Phase 2 ordering in §7), so both are accepting
client connections before `ha_promote` resolves the split-brain. Conflicting
client operations (job submissions, queue modifications) submitted to both
primaries during this window may diverge. This falls within the parent design's
acknowledged `split_brain_risk_accepted` model, with the caveat that the
window is longer than a single heartbeat interval during cold start.
**Data divergence during split-brain**: The losing primary's WAL is discarded when it
steps down to standby — any client operations it accepted during the split-brain
window are lost. This design does not add a reconciliation protocol; the scope is
bounded to the existing HA model. Operators who cannot tolerate any data loss during
cold start should not submit client operations until the primary's `INFO("Promoted
to primary")` log message confirms Phase 2 is complete. The typical cold-start
window occurs only during initial cluster boot or full-cluster restart — not during
normal single-node failover where an existing primary with a term > 0 prevents the
split-brain scenario from occurring.
**PRNG**: The jitter calculation uses an xorshift64 PRNG seeded from
`/dev/urandom`. The PRNG state is stored in `ha_state_t.jitter_seed` (one per
node, initialised during Phase 1). The xorshift64 implementation should be a
shared inline function in `ha.h` (or a small `prng.h` utility header) so both
the election probe (§9.1) and the heartbeat listener (parent HA design) use the
same generator without code duplication. The seed is initialised from
`/dev/urandom` (read 8 bytes) during Phase 1.
If `/dev/urandom` is unavailable, fall back to
`clock_gettime(CLOCK_MONOTONIC, &ts)` nanoseconds XOR'd with
`((uint64_t)priority << 56) ^ ((uint64_t)getpid() << 16)` with a `WARN` log.
The shift mixing avoids the correlation problem where two systemd-started
processes in the same nanosecond produce jitter values differing only by
their priority index (a small integer). The `time(NULL) ^ getpid()` fallback
is rejected because it produces nearly identical values for processes started
simultaneously by systemd (same second, sequential PIDs) — exactly the scenario
where symmetry breaking matters most.
### 9.1 Algorithm (design doc §5.4)
```c
int am_i_primary(cluster_t *c)
{
/* Priority 0: unconditionally primary — highest-priority node does not probe.
* Safe because all controllers receive the same --peers list, so ordering is
* deterministic and priority-0 is unique across the cluster. */
if (c->ha.my_priority == 0) return 1;
/* Launch one pthread per candidate with priority < my_priority.
* Each thread makes a non-blocking connect + poll loop to the candidate,
* sends OP_heartbeat. If ANY thread gets a live ACK before the shared
* election_timeout_ms deadline → return 0 (standby).
* If the deadline expires with no ACK → return 1 (primary). */
atomic_int election_done = 0;
probe_ctx_t probes[MAX_HA_CANDIDATES];
int n_probes = 0;
for (int i = 0; i < c->ha.my_priority; i++) {
probes[n_probes++] = (probe_ctx_t){
.candidate = &c->myconf->peers_cfg.candidates[i],
.ssl_ctx = c->ha.ssl_ctx, /* stashed in Phase 1 */
.timeout_ms = c->ha.config.probe_connect_timeout_ms,
.election_done = &election_done,
.got_ack = 0, /* set to 1 by probe_thread on success */
};
pthread_create(&probes[n_probes-1].thread, NULL,
probe_thread, &probes[n_probes-1]);
}
/* Wait for first ACK or deadline (with jitter for simultaneous-boot, see §9.0) */
long to_ms = c->ha.config.election_timeout_ms;
to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms;
to_ms += (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms);
struct timespec deadline;
clock_gettime(CLOCK_MONOTONIC, &deadline);
deadline.tv_sec += to_ms / 1000;
deadline.tv_nsec += (to_ms % 1000) * 1000000L;
if (deadline.tv_nsec >= 1000000000L) {
deadline.tv_sec++;
deadline.tv_nsec -= 1000000000L;
}
int got_ack = wait_any_probe(probes, n_probes, &deadline);
/* Signal all threads to stop (no pthread_cancel — unsafe through OpenSSL) */
atomic_store(&election_done, 1);
for (int i = 0; i < n_probes; i++)
pthread_join(probes[i].thread, NULL);
return got_ack ? 0 : 1;
}
```
### 9.2 Thread coordination
`pthread_cancel()` is **NOT** used (cancellation through OpenSSL shutdown handlers
can deadlock). Each probe thread uses **non-blocking `connect()` + `poll()`**:
1. `socket()` + `fcntl(O_NONBLOCK)` + `connect()` → expect `EINPROGRESS`
2. `poll()` with timeout `probe_connect_timeout_ms` (default 500ms)
3. Between poll attempts, check `atomic_load(&election_done)` — if set, exit cleanly
4. If `poll()` returns -1 and `errno != EINTR`, treat as probe failure — log at TRACE
level and exit thread with `*got_ack = 0`
5. After connect, `SSL_connect()` with `SSL_ERROR_WANT_READ`/`WANT_WRITE` polling;
on hard SSL failures (certificate expired, protocol mismatch, peer disconnect),
treat as probe failure — log at TRACE level and exit thread
6. If `send()` or `recv()` fails on the SSL socket, treat as probe failure
7. Send `OP_heartbeat`, receive ACK, set `*got_ack` via shared pointer
The per-attempt poll timeout bounds the worst-case exit delay to one poll interval
per thread (~500ms). TCP connect timeout (kernel SYN retry) is not reached because
the thread checks `election_done` between each poll iteration and exits early.
Probe failures are expected when peers are down — only TRACE-level logging is
appropriate; ERROR logs would generate noise during normal operation.
### 9.3 Election outcome logging
```
/* Higher-priority candidate alive: */
INFO("HA election: candidate %s:%u responded — starting as standby (my_priority=%d)",
winner_host, winner_port, my_priority)
/* No higher-priority candidate alive: */
INFO("HA election: no higher-priority candidate responded within %ums — "
"starting as primary (my_priority=%d)",
election_timeout_ms, my_priority)
```
> **SUPERSEDED.** This section (originally: the pthread-based priority
> election — `am_i_primary()`, `probe_thread()`, `wait_any_probe()` — plus the
> pre-iomgr `start_heartbeat_listener()`/`stop_heartbeat_listener()` heartbeat
> listener and the `ha_check_heartbeat_nocluster()` priority-based step-down
> used together for simultaneous-boot split-brain avoidance) described a
> design that was later replaced. Every function named above has been deleted
> from the codebase — see the deletion comments in `src/controller/ha.c`
> (`am_i_primary() deleted — replaced by ha_start_election_async
> (election-iomgr-refactor)`; `start_heartbeat_listener() / stop_heartbeat_listener()
> deleted — never called`; `ha_check_heartbeat_nocluster() / ha_watchdog_thread() /
> ha_start_watchdog() / ha_stop_watchdog() deleted — watchdog pthread replaced
> by ha_arm_heartbeat_timeout_vtask`) and `src/controller/ha.h`. Election is
> now a Raft-style quorum protocol (`ha_start_election_async()` →
> `raft_start_election()`) driving a 4-state FSM (`CTRL_INIT` /
> `CTRL_ELECTING` / `CTRL_PRIMARY` / `CTRL_STANDBY`), with `EPOLLRDHUP`-based
> failover detection replacing the heartbeat-timeout watchdog pthread. See
> "src/docs/config-replication-design.md" findings and the closed specs in
> `.kiro/specs/` — `ha-config-bootstrap`, `election-iomgr-refactor`,
> `raft-library`, `raft-ha-core-safety`, `ha-prevote` — for what shipped
> instead.
---
@ -2098,16 +1873,16 @@ See `FUTURE_WORK.md`.
| File | Change | Complexity |
|---|---|---|
| `src/schema/controller.capnp` | Add `replConfig @29`, `replConfigPush @30`, `upgradeConfig @31` to `OPCode`; add `ReplConfigRequest`, `ReplConfigReply`, `ReplConfigPushRequest`, `ReplConfigPushReply`, `UpgradeConfigRequest`, `UpgradeConfigReply` structs; add `configGeneration @20 :UInt64` to `ClusterConfig` (§4.4); add `sender_priority @0 :UInt32` to `Heartbeat` struct (populated by heartbeat listener, compared by `ha_check_heartbeat_nocluster()` — hard prerequisite for simultaneous-boot symmetry, §9.0); add union arms to `Request` and `RequestReply` | Low |
| `src/schema/controller.capnp` | Add `replConfig @29`, `replConfigPush @30`, `upgradeConfig @31` to `OPCode`; add `ReplConfigRequest`, `ReplConfigReply`, `ReplConfigPushRequest`, `ReplConfigPushReply`, `UpgradeConfigRequest`, `UpgradeConfigReply` structs; add `configGeneration @12 :UInt64` to `ClusterConfig` (§4.4; implemented at field number @12, not the @20 originally planned at design time); add `sender_priority @0 :UInt32` to `Heartbeat` struct (populated by heartbeat listener, compared by `ha_check_heartbeat_nocluster()` — hard prerequisite for simultaneous-boot symmetry, §9.0); add union arms to `Request` and `RequestReply` | Low |
| `src/controller/main.c` | Update `conf_file` resolution: read `conf/current` pointer file, then open named versioned file (replaces flat `conf/current_config` path); remove stack-allocated `char conf_file[]` and the `myconf.conffile = conf_file` assignment — `conffile` is now heap-allocated via `asprintf()` in `load_primary_config()` or `fetch_config_from_primary()` and freed in `myconf_fini()`; add `--conf-stale-timeout` and `--conf-keep-n` CLI flag parsing; add `--peers` validation (WARN on empty host or port 0) | Trivial |
| `src/include/myconf.h` | Add `config_version_tag` (uint64_t), `config_sha256[32]`, `conf_keep_n`, `config_generation` (uint64_t, §4.4), `conf_stale_timeout` (time_t, default 86400), and `config_stale` (bool) fields to `myconf_t` | Trivial |
| `src/include/xce/cluster_def.h` | Add `restart_requested` (atomic_int, for config-change restart), `heartbeat_listener_tid`, `heartbeat_listener_running`, and `lost_and_found_job_count` (atomic_int, incremented during `persist_recover()` when jobs are routed to the synthetic queue, §6.5) fields to `cluster_t` | Trivial |
| `src/controller/myconf.c` | Replace `get_default_conf()` with `load_primary_config()` (fail-fast + SHA-256 compute on primary path); config save/trim/validation functions moved to `config_repl.c`; add `myconf_fini()` to free `conffile` (now heap-allocated via `asprintf()` per §3.2), `cluster_config`, `config_version_tag`, and `config_sha256` | Low |
| `src/controller/cluster.c` | Split `cluster_init()` into phase 1 and phase 2; gate config load on role; move `iomgr_init` and `endpoint_init` to phase 2; add `create_lost_and_found()` call before `persist_recover`; add `OP_replConfig`, `OP_replConfigPush`, and `OP_upgradeConfig` cases to `myreq_handler()` opcode dispatch; `cluster_run()` event loop checks `c->restart_requested` and exits cleanly when set; `cluster_fini()` joins heartbeat listener thread on any exit path and frees `standby_endpoints` | Medium |
| `src/controller/myapp.c` | Restructure `myapp_run()`: call phase 1, then `am_i_primary()`, then config load/fetch, then phase 2. When `cluster_run()` returns cleanly (restart_requested=1), `myapp_run()` returns 0 to main() which exits; the systemd service manager restarts the process. Call `myconf_fini()` in the cleanup path before `cluster_fini()` | Medium |
| `src/controller/config_repl.c` | **Phase 1**: `conf_read_current_filename()` (read HEAD-style pointer), `conf_read_raw()` (read + size-check, enforcing `CONF_MAX_SIZE`), `conf_save_config()` (write + fsync + thread-unique pointer-file swap + trim), `conf_trim_old_versions()`, `compute_config_sha256()`, `check_conf_stale()` (mtime-based staleness check, §10.3), `sanitise_cn()` (cert CN log sanitisation, §6.4), `config_contains_reserved_name()` (lost_and_found name collision check, §7.1). **Phase 2**: `handle_upgrade_config()` (SHA-256 verify, Cap'n Proto decode, generation increment §4.4, re-encode with gen+1, save, push), `push_config_to_standbys()`, `create_lost_and_found()` (synthetic queue + host, §6.5), `resolve_queue_or_lost_and_found()`, `resolve_host_or_lost_and_found()`, `clear_resource_and_pend()`. Compile-time gate: `CONF_ENABLE_UPGRADE_CONFIG` (default 0, §5.3) | Medium |
| `src/controller/config_repl.c` | **Phase 1**: `conf_read_current_filename()` (read HEAD-style pointer), `conf_read_raw()` (read + size-check, enforcing `CONF_MAX_SIZE`), `conf_save_config()` (write + fsync + thread-unique pointer-file swap + trim), `conf_trim_old_versions()`, `compute_config_sha256()`, `check_conf_stale()` (mtime-based staleness check, §10.3), `sanitise_cn()` (cert CN log sanitisation, §6.4), `config_contains_reserved_name()` (lost_and_found name collision check, §7.1). **Phase 2**: `handle_upgrade_config()` (SHA-256 verify, Cap'n Proto decode, generation increment §4.4, re-encode with gen+1, save, push), `push_config_to_standbys()`, `create_lost_and_found()` (synthetic queue + host, §6.5), `resolve_queue_or_lost_and_found()`, `resolve_host_or_lost_and_found()`, `clear_resource_and_pend()`. Compile-time gate: `CONF_ENABLE_UPGRADE_CONFIG` (default 0 as designed; ships hardcoded to 1, §5.3 drift note) | Medium |
| `src/include/byteorder.h` | **New file**: `be64toh()` fallback macro for non-glibc or pre-2.25 targets (§6.4 portability note). The Meson build checks for `be64toh()` at configure time; if unavailable, this header provides a portable implementation via `__builtin_bswap64()` or `le64toh()`/`ntohll()` | Trivial |
| `src/controller/meson.build` | Add `-DCONF_PUSH_TIMEOUT_MS=500` and `-DCONF_ENABLE_UPGRADE_CONFIG=0` compile definitions (defaults, overridable via Meson options); add `config_repl.c`, `config_repl.h`, `byteorder.h` to sources | Trivial |
| `src/controller/meson.build` | Add `-DCONF_PUSH_TIMEOUT_MS=500` and `-DCONF_ENABLE_UPGRADE_CONFIG=0` compile definitions (defaults, overridable via Meson options); add `config_repl.c`, `config_repl.h`, `byteorder.h` to sources. **Not implemented as designed**: both macros are defined via `#ifndef`/`#define` fallback directly in `config_repl.h` instead (with the enable flag hardcoded to `1`) — `meson.build` has no `-Denable-upgrade-config` option and no override of either macro. | Trivial |
| `src/controller/ha.c` | Replace `am_i_primary()` stub with real election probing; add `probe_thread()`, `wait_any_probe()`; add `start_heartbeat_listener()` / `stop_heartbeat_listener()`; add `xorshift64_next()` (or use shared `ha.h` inline); enhance `ha_check_heartbeat_nocluster()` with priority comparison for simultaneous-boot symmetry breaking (§9.0) | High |
| `src/controller/ha.h` | Add `ssl_ctx` field to `ha_state_t` (use `void*` to avoid pulling in `<openssl/ssl.h>` — cast to `SSL_CTX*` at use sites in election probe and heartbeat listener); add `jitter_seed` (uint64_t, for xorshift64 PRNG) and `xorshift64_next()` inline function to `ha_state_t`; add `zarray_t *standby_endpoints` to `ha_state_t` for tracking connected standby controllers; add `probe_ctx_t` struct; add `election_jitter_max_ms` and `election_priority_step_ms` to `ha_config_t` with defaults `HA_DEFAULT_ELECTION_JITTER_MAX_MS 200` and `HA_DEFAULT_ELECTION_PRIORITY_STEP_MS 500` (§9.0 priority-based base delay); update "no new TLS" comment to note SSL transport is sufficient for trusted-LAN model; update `am_i_primary()` declaration; declare `ha_is_primary()` for runtime role check (§10.12) | Low |
| `src/controller/join.c` | Update `handle_controller_join()`: remove `-ENOSYS` (controller-controller sessions are valid — config exchange is dispatched from `myreq_handler()`); register connected standby endpoints in `c->ha.standby_endpoints` after successful join handshake; handle endpoint removal on disconnect | Low |
@ -2123,7 +1898,8 @@ Note: the existing `src/controller/repl.c` (WAL replication — standby slot man
ACK tracking, catch-up) is a separate module with a similar name but different domain.
`repl.c` uses `replicator_t.standbys[]` for per-connection WAL ACK tracking;
`config_repl.c` uses `ha_state_t.standby_endpoints` for config push fan-out.
See `src/controller/repl.c` (340 lines) for the existing WAL replication module.
See `src/controller/repl.c` (881 lines as of this audit, up from 340 at design time)
for the existing WAL replication module.
**Total estimated LOC**: ~510 net new lines across all files (updated from ~480 to
account for compile-time gate, `sanitise_cn()`, `config_contains_reserved_name()`,

47
src/docs/config-replication-impl-status.md
View file

@ -29,28 +29,40 @@ of 2026-05-22 (commits `2ed7e34`–`5053c8d`).
| Raw capnp shims (`request.c`) | ✅ Done | `26393e9` | `xce_replconfig_reply_raw()`, `xce_replconfig_req_ssl()`, `xce_replconfig_reply_ssl_decode()`, `xce_upgrade_config_req_extract_raw()` — bypass `xce_data_t` for raw bytes |
| `handle_config_request()` (primary handler) | ✅ Done | `4ef2089` | Tag compare, SHA-256 double-check; sends real bytes via `xce_replconfig_reply_raw()` |
| `fetch_config_from_primary()` + `do_one_config_fetch()` | ✅ Done | `2aae45f` | Full SSL roundtrip: non-blocking connect+poll, `xce_replconfig_req_ssl()`, SHA-256 verify, `conf_save_config()`, reload; 20-attempt exponential backoff |
| `handle_controller_join()` | ✅ Done | `8f4db30` | Registers endpoint in `standby_endpoints` under lock |
| `handle_controller_join()` | ⚠️ Reversed | `8f4db30` → `40f510fb` | Originally registered the endpoint in `standby_endpoints` under lock, but this was later identified as a live ownership-model violation (controller-controller peering is primary-initiated outbound only, per `ha-async-io.md`'s Connection Ownership). `40f510fb` (post-dates this doc) rewrote it to unconditionally reject inbound `role=HostRole_controller` joins with `-E_XCE_BAD_ROLE`; it no longer touches `standby_endpoints` |
| WAL record-count `schedule_checkpoint()` | ✅ Done | `a95476b` | Idempotent CAS; coalesced via `delaymgr` 1s delay; bounds `persist_recover` to ≤10k records |
| `typemap.h` union arms + capnp shims | ✅ Done | `491c255` | `replconfigreplymsg`, `upgradeconfigreplymsg`, etc. wired |
| API server rejects `lost_and_found` job submissions | ✅ Done | `bc92166` | `preCheckJobRequest()` returns "internal recovery queue" error before store lookup; 3 tests in `jobs_test.go` |
---
## Phase 2 (Remote config push) — COMPILED, GATED
## Phase 2 (Remote config push) — COMPILED, ENABLED BY DEFAULT
Phase 2 infrastructure ships compiled but inert. The `CONF_ENABLE_UPGRADE_CONFIG`
compile-time gate (default: 0) prevents `OP_upgradeConfig` from being reachable
in production builds until both prerequisites are met (see §5.3 and §2 of the
design doc).
> **⚠ Doc drift note (2026-08-08):** the paragraph and prerequisites list below
> describe Phase 2 as gated off pending two prerequisites. Neither prerequisite
> is met, but `CONF_ENABLE_UPGRADE_CONFIG` has defaulted to **1** (enabled)
> since commit `fd93997f` (2026-05-31) — 9 days after this doc's own "updated"
> date — and has never been reverted. `OP_upgradeConfig` is a **live, reachable
> path in current production builds**. See `src/controller/config_repl.h:27-29`
> and dispatch at `src/controller/cluster.c:772-807`. This is a real,
> outstanding risk, not a future gate — track via `.kiro/steering/roadmap.md`.
Phase 2 infrastructure ships compiled and **enabled**. The
`CONF_ENABLE_UPGRADE_CONFIG` compile-time gate (default: **1** as of
`fd93997f`, 2026-05-31; originally 0) was intended to keep `OP_upgradeConfig`
unreachable in production builds until both prerequisites below are met (see
§5.3 and §2 of the design doc), but the gate now defaults to enabled with
neither prerequisite satisfied.
| Component | Status | Notes |
|---|---|---|
| `handle_upgrade_config()` | ✅ Wired, gated | `5053c8d` — pre-decode opcode peek in `myreq_handler()` captures raw packet bytes; `xce_upgrade_config_req_extract_raw()` extracts sha256+bytes; delegates to `do_upgrade_config_with_bytes()`. Active only when `CONF_ENABLE_UPGRADE_CONFIG=1` |
| `handle_upgrade_config()` | ✅ Wired, enabled by default | `5053c8d` — pre-decode opcode peek in `myreq_handler()` captures raw packet bytes; `xce_upgrade_config_req_extract_raw()` extracts sha256+bytes; delegates to `do_upgrade_config_with_bytes()`. Reachable by default since `CONF_ENABLE_UPGRADE_CONFIG` defaults to 1 (`fd93997f`, 2026-05-31) |
| `push_config_to_standbys()` | ✅ Timeout enforced | `f3babec` — `wait_all_with_timeout()` with `CONF_PUSH_TIMEOUT_MS` (500ms) aggregate deadline. Push fan-out wire protocol still best-effort; standbys catch up via pull on reconnect |
| `create_lost_and_found()` | ✅ Done | `91e66b8` | Inserts synthetic queue via `create_queue()` and host via `makeWorker()` before `persist_recover()`; gated with `#ifndef XCE_TESTING` for unit test binary |
**Prerequisites before enabling** (`CONF_ENABLE_UPGRADE_CONFIG=1`):
1. `validate_host()` implemented with real CN allowlist check (see `FUTURE_WORK.md §6`)
**Prerequisites — NOT met, despite the gate defaulting to enabled**
(`CONF_ENABLE_UPGRADE_CONFIG=1` in `src/controller/config_repl.h:27-29`):
1. ❌ `validate_host()` still an unconditional-accept stub — `src/controller/join.c:13-16` returns `1` for every host/key pair, i.e. no real CN allowlist check (see `FUTURE_WORK.md §6`, confirmed still open per `.kiro/steering/roadmap.md:82-83`)
2. Admin config tool exists and tested end-to-end (see `FUTURE_WORK.md §1`)
---
@ -59,14 +71,21 @@ design doc).
| Suite | Tests | Covers |
|---|---|---|
| `test_config_repl` | 24 | Storage, SHA-256, trim, stale, sanitise_cn, path consistency, push timeout, capnp shim roundtrips |
| `test_config_repl` | 101 (was 24 as of this doc's date) | Storage, SHA-256, trim, stale, sanitise_cn, path consistency, push timeout, capnp shim roundtrips — grown substantially since 2026-05-22 by `ha-repl-hardening`, `ha-term-safety`, `raft-adapter-conformance`, `config-upgrade-snapshot`, and assorted bug-fix coverage (e.g. bug-1475, bug-1476) |
| `test_am_i_primary` | 3 | Priority-0 immediate, priority-0 with peers, deadline-expire |
| `test_ha_role` | (existing) | Role transitions, watchdog, redirect |
| `test_persist_wal` | 6 (+3 new) | WAL append, CAS idempotency, `max_wal_records` default |
| `test_ha_cluster` | 5 | Cluster HA integration |
| `test_p1_integration` | 5 | Phase 1 init structure |
| All other suites | ~41 | 0 failures |
| **Total** | **84** | **0 failures, 0 ignored** |
| **Total** | **84 at doc date; stale — `test_config_repl` alone now exceeds this** | **0 failures, 0 ignored (as of 2026-05-22)** |
> **Note (2026-08-08):** this table was accurate as of the doc's 2026-05-22
> date but has not been kept in sync with ~2.5 months of subsequent test
> growth. `test_config_repl` alone now has 101 `RUN_TEST(...)` registrations
> (verified via `src/controller/testsuite/test_config_repl.c`), so the suite
> total is understated. Re-run and re-tally before relying on this table for
> current coverage figures.
---
@ -74,10 +93,10 @@ design doc).
| Item | Severity | Tracked |
|---|---|---|
| Heartbeat listener (`start_heartbeat_listener`) is a no-op stub — simultaneous-boot symmetry relies on priority-step delay only | LOW | `FUTURE_WORK.md §7` |
| ~~Heartbeat listener (`start_heartbeat_listener`) is a no-op stub~~ — **RESOLVED/OBSOLETE**: `start_heartbeat_listener()`/`stop_heartbeat_listener()` have been deleted from the codebase entirely (never called, per `election-iomgr-refactor`; see `src/controller/ha.h:798`, `src/controller/ha.c:2916`). The priority-step-delay election model this limitation described has been superseded by real Raft-based election/prevote machinery (`raft-library`, `ha-prevote`, `election-iomgr-refactor` specs, all CLOSED) | — | superseded |
| Peers cross-validation hash in `OP_heartbeat` not implemented | LOW | `FUTURE_WORK.md` |
| `OP_replConfigPush` standby-side handler not implemented — pushed config accepted only via pull on reconnect | LOW | `FUTURE_WORK.md` |
| WAL routing helpers (`resolve_queue_or_lost_and_found` etc.) not wired in `apply_wal_entry()` | LOW | `FUTURE_WORK.md` |
| ~~`OP_replConfigPush` standby-side handler not implemented~~ — **FALSE, now implemented**: `handle_repl_config_push()` (`src/controller/config_repl.c:1308`) is fully implemented — validates the request, checks SHA-256 (line 1341), calls `conf_save_config()` (line 1357), activates the config, and sends an explicit `ReplConfigPushReply`. Dispatched from `OPCode_replConfigPush` in `src/controller/cluster.c:754-757`. Added by commit `3d076ccd` ("fix(ha): fix config replication to standbys; add replConfigPush handler"), which post-dates this doc | — | resolved |
| WAL routing helpers (`resolve_queue_or_lost_and_found` etc.) not wired in `apply_wal_entry()` — **unverifiable as written**: `resolve_queue_or_lost_and_found` no longer appears anywhere in `src/controller/*.c`/`*.h`. `apply_wal_entry()` (`src/controller/persist.c:3281`) and the `WAL_CONFIG_MARKER` apply path (`persist.c:3851-3866`, added by `config-upgrade-snapshot` task 5) now treat lost-and-found entity re-homing as a deliberately separate, later step from apply/activation — suggesting this routing design was reworked or renamed rather than simply left unfinished. Needs re-investigation, not a straight carry-forward | LOW | `FUTURE_WORK.md` |
---

51
src/docs/deployment.md
View file

@ -7,11 +7,11 @@ Covers building and running all four daemons (controller, worker, scheduler, api
```
xceadmin (Python CLI) --config push (TLS)--> controller (C, HA cluster, 1..N nodes)
^ ^
plaintext join | | plaintext join
TLS join | | TLS join
| |
worker (C) scheduler (C)
^
CGO-linked libxce client (TCP, HA-aware)
CGO-linked libxce client (TCP, TLS, HA-aware)
|
apiserver (Go, REST) <-- xcecli (Go CLI, REST)
```
@ -23,16 +23,23 @@ xceadmin (Python CLI) --config push (TLS)--> controller (C, HA cluster, 1..N n
- **xcecli** — Go CLI, talks to `apiserver` over REST, never touches the controller directly.
- **xceadmin** — Python CLI, the only tool that compiles and pushes *cluster configuration* (`cluster.bin`) to the controller. Separate concern from the four daemons above.
Only controller↔controller HA replication traffic is TLS. Worker→controller, scheduler→controller, and apiserver→controller traffic is **plaintext**.
**Every** connection into the controller is TLS-encrypted — controller↔controller HA replication, and every `libxce`-client connection (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller). The controller runs a single `iomgr` bound to one server `SSL_CTX` (`myapp.c`); `endpoint_accept()`/`endpoint_connect()` in `src/lib/netio.c` have no plaintext code path at all — every accepted or outgoing socket is wrapped in `SSL_new()` before use. What's actually missing is **client authentication and identity verification**, not encryption: TLS verification is `SSL_VERIFY_NONE` everywhere in this codebase (server and client contexts alike, no client certificates), so any of these connections buys you encryption-in-transit, not authentication of who's on the other end.
## Building
Two different Nix entry points, for two different jobs — don't conflate them:
- **`nix develop --command` — single-component dev builds.** `/xce-build` is not a separate build system; it's `nix develop ./src --command bash src/tools/xce-build.sh <component> <profile>` under the hood, i.e. the hermetic dev shell driving meson/ninja for exactly one component. This is the reproducible way to build just `controller`, just `worker`, etc.
- **`nix build` — whole-stack production packaging.** A packaged Nix derivation, all-or-nothing (see Production Gaps item 6) — it always produces every daemon in one `result/`, never a single one.
```bash
# Dev loop, per component (debug + ASAN, runs tests)
# Dev loop, per component (debug + ASAN, runs tests) — wraps `nix develop ./src --command ...`
/xce-build lib # libxce only
/xce-build controller
/xce-build worker
/xce-build scheduler
/xce-build apiserver
/xce-build apiserver # also builds xcecli in the same pass (same meson step)
/xce-build xcecli # xcecli alone, if you don't need apiserver
/xce-build xceadmin
/xce-build # all components
@ -41,10 +48,10 @@ Only controller↔controller HA replication traffic is TLS. Worker→controller,
# debug -> build.dbg/ -O0 -g3, no sanitizer, gdb-friendly
# release -> build.release/ -O3, no sanitizer, build-only
# Production (Nix)
# Production packaging (Nix, whole stack only)
nix build .#xcecore # controller + worker + scheduler + apiserver + xceadmin + AppImage, one derivation
nix build .#xcecore_debug # same, debug + ASAN (this flake's default package)
nix build .#xcecli # xcecli only (separate Go-only derivation)
nix build .#xcecli # xcecli, as its own separate Nix *package* derivation (Go-only, no C toolchain) — `/xce-build xcecli` above gets you the same binary without going through `nix build` at all
```
`nix build .#xcecore` is all-or-nothing — there is no per-daemon Nix target. It produces one `result/bin/` containing `xce_controller`, `xcecd` (+ `xcecd_worker`, `xcecd_virtual`), `xce_scheduler`, `xce_apiserver`, the `xceadmin` venv wrapper, and `xce` (the AppImage dispatcher script, see below). `nix build .#xcecli` is the one genuinely separate target.
@ -96,8 +103,8 @@ xceadmin config push --dry-run cluster.bin --controller 127.0.0.1:9991
### Known limitations (as of this writing)
- **`config push` against a real controller fails with a client-side decode crash**, live-verified against a freshly deployed single-node controller: the controller's `OPCode_upgradeConfig` dispatch is fine — `CONF_ENABLE_UPGRADE_CONFIG` defaults on, the earlier dispatch-gate polarity bug (bug-1468) was already fixed 2026-07-25, and the controller log shows the push genuinely committed, activated, and hash-verified server-side. The bug is in `xceadmin` itself: `xce_reply()` (`src/lib/request.c`) wraps every reply — including the upgradeConfig reply — in an outer `Request{opcode: OPCode_reply, body.replymsg: RequestReply{...}}` envelope, but `discovery.py`'s `_redirect_target()` decodes the raw wire bytes directly as a bare `RequestReply`, skipping the outer envelope. The resulting pointer-offset misalignment throws `capnp.lib.capnp.KjException: ... non-list pointer where text was expected` from `push_config()`. This affects every opcode reply routed through `follow_redirects`, not just upgradeConfig. Fix belongs in `xceadmin`, not the controller.
- **`config push --dry-run`'s "OK" result is not meaningful evidence of anything**: `probe_config()`'s docstring itself notes `OPCode_probeConfigTag` has no dispatch case in `cluster.c`, so the "accepted" response is a client-side inference, not a real wire signal from the controller. Live-verified: dry-run returns exit 0 with no corresponding entry in the controller log at all.
- **~~`config push` against a real controller fails with a client-side decode crash~~** — **Fixed 2026-07-30.** `decode_reply()` (`src/xceadmin/src/xceadmin/discovery.py:46-77`) now unwraps the outer `Request{opcode: OPCode_reply, body.replymsg: RequestReply{...}}` envelope before decoding the inner `RequestReply`. Both `_redirect_target()` and `push_config()` go through `decode_reply()`; the old bare-`RequestReply` decode path is gone. `decode_reply()` validates `body.which() == "replymsg"` and raises `ReplyDecodeError` on anything else. Live-verified against a real deployed controller.
- **~~`config push --dry-run`'s "OK" result is not meaningful evidence of anything~~** — **Fixed 2026-08-09 (bug-1506/1507).** The controller gained a real `OPCode_probeConfigTag` handler (`config_repl.c:handle_probe_config_tag`) that returns the current `config_version_tag` through the standard `Request`/`RequestReply` envelope. `probe_config()` now wraps the probe in a `Request` envelope and goes through `follow_redirects` → `decode_reply` (same as `push_config`), so `accepted=True` means the controller actually replied with a valid tag, and `--dry-run` reports the real generation number from the cluster.
- No `config diff`, no rollback, no version history command. `xceadmin` today has exactly two subcommands: `config build` and `config push [--dry-run]`.
- No client authentication — the trust model is "same posture as controller-to-controller TLS," i.e. encryption without identity verification.
@ -117,7 +124,7 @@ xce_controller \
- Omit `--peers` for a single-node, non-HA controller.
- `--workdir` holds per-node `conf/` (cluster config) and `data/` (WAL + checkpoint) subdirectories.
- `--certdir` must contain `xce.crt`/`xce.key` — this is the **only** TLS material in the whole stack, used exclusively for controller↔controller HA traffic. Generate a self-signed pair for local/dev use:
- `--certdir` must contain `xce.crt`/`xce.key` — this is the **only** TLS material in the whole stack, and the controller loads it into the single server `SSL_CTX` used for **every** inbound connection it accepts: controller↔controller HA traffic *and* every worker/scheduler/apiserver/xceadmin client join. Generate a self-signed pair for local/dev use:
```bash
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
-keyout /var/lib/xce/certs/xce.key -out /var/lib/xce/certs/xce.crt -subj "/CN=xce"
@ -163,7 +170,7 @@ xcecd -m controller-host -p 9990 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990" -l /var
log_dir: "/var/log/xce"
```
- `--peers "[name=]host:port,..."` enables HA-aware controller discovery: the worker tries each candidate in turn (every 5s by default) until one accepts the join.
- No TLS on this connection — worker↔controller traffic is plaintext.
- Worker↔controller traffic is TLS-encrypted (via the shared `libxce` client, `xcecli_ctx_init()` → `xce_ctx_init()`), same posture as everything else in the stack: no client certificate, `SSL_VERIFY_NONE`.
## Deploying the Scheduler
@ -174,7 +181,7 @@ xce_scheduler --peers "ctrl-1=controller-host:9990" -l /var/log/xce/scheduler.lo
- No config file at all — pure CLI (`argp`).
- **Gotcha**: the default controller target is hard-coded to `localhost:9990`. There is no `-m`/`-p` equivalent — the *only* way to point the scheduler at any other host or port, even a single non-HA controller, is `--peers`. Forgetting `--peers` silently targets `localhost:9990` regardless of where your controller actually is.
- `-v` is repeatable for more verbose logging; default log file is `/tmp/xce_scheduler.log`.
- No TLS — plaintext to the controller, same as the worker.
- TLS-encrypted to the controller, same as the worker (same shared `libxce` client path).
- Scheduling policies (fcfs, limit, parallel/sequential-job allocators) are compiled into the binary through the picker plugin interface (`builtin_fcfs` is the registered default), not dynamically loaded.
### Session recording (live cluster)
@ -241,7 +248,7 @@ Both `app.yaml` and `logger.yaml` are JSON-Schema-validated at startup (fail-fas
### Relationship to the controller
apiserver is a **separate OS process**, network-connected to the controller — not embedded, not in-process. It joins the controller's cluster protocol as `HostRole_apiserver` via a CGO-linked `libxce` client (the same join mechanism workers/schedulers use), configured entirely by `app.yaml`'s `backend:` block. If `backend.address` is empty, apiserver logs "running in stub mode" and serves HTTP, but every job/host/queue/cluster handler will fail. No TLS on this connection.
apiserver is a **separate OS process**, network-connected to the controller — not embedded, not in-process. It joins the controller's cluster protocol as `HostRole_apiserver` via a CGO-linked `libxce` client (the same join mechanism workers/schedulers use), configured entirely by `app.yaml`'s `backend:` block. If `backend.address` is empty, apiserver logs "running in stub mode" and serves HTTP, but every job/host/queue/cluster handler will fail. This connection is TLS-encrypted, same as every other `libxce` client (no client certificate, `SSL_VERIFY_NONE`) — do not confuse it with apiserver's own separate HTTPS termination for its REST API, covered next.
### TLS
@ -277,18 +284,18 @@ xceadmin config push cluster.bin --controller localhost:9991
Be aware of these before treating anything above as a production runbook — they are honest gaps in this repo today, not documented-but-hidden features:
1. **No process supervision is shipped.** No systemd unit, no Dockerfile, no docker-compose, no supervisord config exists anywhere in this repo for any of the four daemons. A systemd-unit *sketch* exists in `src/docs/config-replication-design.md` but has a wrong binary name (`xcecontroller` vs the real `xce_controller`) and specifies `Type=notify`, which the controller does not support (it never calls `sd_notify`) — using it as written will hang at startup and time out. If you adapt it, switch to `Type=simple` and fix the binary name.
2. **`src/frontend/apiserver/config/auth.yaml` ships plaintext dev credentials checked into git**, including an `admin`/`admin` account and a named-user `cluster_admin` account. Treat all of them as exposed by default — don't ship the checked-in file as-is; copy `auth.yaml.example` and set real credentials/an external auth provider instead.
3. **Only controller↔controller traffic is encrypted.** Worker→controller, scheduler→controller, and apiserver→controller are all plaintext, with no path to enable TLS on them today.
4. **apiserver's own TLS uses a checked-in self-signed cert with no CA trust story**; `xcecli` hardcodes `InsecureSkipVerify: true` against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated.
5. **`xceadmin config push` doesn't work against a real cluster yet** (see xceadmin's Known Limitations above) — the controller side is fine; `xceadmin`'s `discovery.py` fails to unwrap the `Request` envelope around every reply, crashing on decode.
6. **`nix build .#xcecore` is all-or-nothing** — there's no way to build just one daemon via Nix; use `/xce-build <component>` for per-component builds.
2. **`src/frontend/apiserver/config/auth.yaml` ships plaintext dev credentials checked into git**: four accounts total — `admin`/`admin` and `op7x2k9`/`CHANGE_ME_dev_only` (both role `cluster_admin`), plus `user1`/`user1` and `user2`/`user2` (role `basic_user`). Treat all of them as exposed by default — don't ship the checked-in file as-is; copy `auth.yaml.example` and set real credentials/an external auth provider instead.
3. **All controller-bound traffic is encrypted but not authenticated.** Controller↔controller HA replication and every `libxce` client (worker→controller, scheduler→controller, apiserver→controller, xceadmin→controller) run over TLS, but `SSL_VERIFY_NONE` is used throughout — no client certificates, no hostname/identity verification. This buys confidentiality and tamper-resistance in transit, not authentication of the peer; anyone who can reach the port and complete a TLS handshake is treated as trusted the moment the transport-level join succeeds.
4. **apiserver's own TLS uses a checked-in self-signed cert with no CA trust story**; `xcecli` hardcodes `InsecureSkipVerify: true` against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated. (This is apiserver's REST-facing HTTPS, separate from its TLS-encrypted `libxce` connection to the controller in item 3.)
5. **`xceadmin config push` against a real cluster** — the decode-crash bug is fixed (2026-07-30, `decode_reply()` unwraps the `Request` envelope), but `--dry-run` still has no real controller-side handler (see Known Limitations item 2).
6. **`nix build .#xcecore` (the packaged derivation) is all-or-nothing** — there's no per-daemon Nix package target. For a single component, use `/xce-build <component>`, which itself runs inside the Nix dev shell (`nix develop ./src --command ...`) rather than through `nix build`.
## Quick Reference
| Daemon | Binary | Config | Controller-pointing flag | TLS |
|---|---|---|---|---|
| Controller | `xce_controller` | none (CLI only) | `--peers` (omit for single-node) | Yes, controller↔controller only |
| Worker | `xcecd` | `worker.yaml` (or CLI) | `-m`/`-p` (config file wins) or `--peers` | No |
| Scheduler | `xce_scheduler` | none (CLI only) | `--peers` (**required** even for a single non-default controller) | No |
| apiserver | `xce_apiserver` | `config/{app,auth,rbac,logger}.yaml` | `app.yaml`'s `backend:` block | Optional, self-terminated |
| Controller | `xce_controller` | none (CLI only) | `--peers` (omit for single-node) | Yes, all inbound (peers + clients), no client cert |
| Worker | `xcecd` | `worker.yaml` (or CLI) | `-m`/`-p` (config file wins) or `--peers` | Yes, no client cert |
| Scheduler | `xce_scheduler` | none (CLI only) | `--peers` (**required** even for a single non-default controller) | Yes, no client cert |
| apiserver | `xce_apiserver` | `config/{app,auth,rbac,logger}.yaml` | `app.yaml`'s `backend:` block | To controller: yes, no client cert. Own REST API: optional, self-terminated |
| xceadmin | `xceadmin` (Python) | none — takes YAML source dir | `--controller host:port` | Yes, no client cert |

3
src/docs/p1-2-host-select.md
View file

@ -33,7 +33,8 @@ The full feature requires two layers:
**Files implemented:**
- `include/resreq.h` — expression-tree types, enums, `extern` declarations.
- `lib/xce_select.peg` — PEG grammar for select boolean expressions.
- `lib/xce_select.c` — `parse_select()`, `eval_select_simple()`, CSE tree.
- `lib/resreq.c` — `parse_select()`.
- `lib/xce_select.c` — `eval_select_simple()`, CSE tree.
- `lib/meson.build` — `xce_select.peg` in build.
### #12 P1-2b: Go EvalSelectForHost + FilterHostByResReqSelect (remaining)

1210
src/docs/persistence-and-ha-design.md
View file

File diff suppressed because it is too large Load diff

274
src/docs/resreq-ast-cse-design.md
View file

@ -1,6 +1,6 @@
# XCE Resource Requirement AST & CSE Design
> **Status**: Draft
> **Status**: Implemented — CSE intern table, GC, flatten/const-fold passes, and grammar are live in `src/lib/xce_select.c` / `src/lib/xce_select.peg`; `sched_meet_host_selection` and GC mark/sweep are wired into the scheduler (`src/scheduler/src/scheduler.c`, `src/scheduler/src/core_session.c`).
> **Last Updated**: 2026-04-27
> **Revision**: 6 (multi-round review: resmgr API, atomic memory order, sel_make_key op2, makeString TOCTOU)
> **Target**: `src/lib/` (C shared library), `src/include/resreq.h`
@ -108,8 +108,9 @@ to them but should not be considered final for those use cases.
- GC idle invariant predicates (`sched_session_active`, `sched_eval_in_progress`,
`staging_merge_complete`) — required to enforce the §5.5 runtime guard that
prevents use-after-free when GC runs. These are tested as stubs in Phase 1
and wired to real `xce_scheduler_t` fields in Phase 6 (see Phase 6 for
the stub-to-production migration contract).
and wired to real scheduler fields in Phase 6 (see Phase 6 for
the stub-to-production migration contract; the fields ended up in
`struct xce_sched_core_` in `core.h`, not `struct xce_scheduler_` — see §5.5).
- live_jobs materialization — iterating `sched->jobs` to produce the flat
`const xce_sched_job_t**` array required by `gc_sweep_select_nodes`.
This is a Phase 6 deliverable, not a standalone library feature.
@ -1073,7 +1074,14 @@ if (sched_session_active(sched) || sched_eval_in_progress(sched)
staging_merge_complete(sched));
return; /* do not sweep — prevent use-after-free */
}
gc_sweep_select_nodes(live_jobs, njobs);
/* Mark phase is a separate, caller-driven step — see the three-function
* protocol below. gc_sweep_select_nodes only sweeps; it does not mark. */
gc_clear_marks();
for (size_t i = 0; i < njobs; i++) {
/* dispatch on live_jobs[i]->resreq->tag and call gc_mark_select_tree()
* on each reachable select root — see the pseudocode below */
}
gc_sweep_select_nodes(NULL, 0);
```
Where `sched_session_active`, `sched_eval_in_progress`, and
@ -1086,13 +1094,18 @@ development and unit testing), these predicates are implemented as thin
wrappers around a `fake_scheduler_t` struct declared in `test_select.c` —
a test-only scaffold with the same three `_Atomic int` fields. In Phase 6
(scheduler wiring), the wrapper bodies are updated to reference the real
`xce_scheduler_t` fields added in that phase. The function signatures do
not change between phases; only the struct pointer inside each wrapper body
changes. No Phase 1 stub code leaks into the production implementation.
scheduler core fields added in that phase (see below for where they ended
up). The function signatures do not change between phases; only the
struct pointer inside each wrapper body changes. No Phase 1 stub code
leaks into the production implementation.
**Phase 6 predicate fields in `xce_scheduler_t`:** Three `_Atomic int`
fields must be added to `struct xce_scheduler_` (in
`src/scheduler/include/scheduler.h`):
**Phase 6 predicate fields:** Three `_Atomic int` fields must be added to
hold this state. As implemented, they live in `struct xce_sched_core_`
(in `src/scheduler/include/core.h`) rather than in `struct xce_scheduler_`
in `scheduler.h` — the later scheduler-core core/driver-split refactor
(`.kiro/specs/scheduler-core/`) relocated the scheduler's session-lifecycle
state into a dedicated `xce_sched_core_t` embedded in the scheduler, and
these predicate fields moved with it:
```c
_Atomic int session_active; /* 1 during scheduling session, 0 between */
@ -1100,60 +1113,84 @@ _Atomic int eval_in_progress; /* 1 while eval_select loop is running, 0 other
_Atomic int staging_merge_complete; /* 1 after staging merge completes, 0 before */
```
These fields are initialized to 0 in the scheduler's init function
(Phase 6 wiring) and set/cleared at the appropriate session-loop
transitions. The predicate wrappers read them via
`atomic_load_explicit(..., memory_order_acquire)` to ensure visibility
from the allocator thread's writes. The `fake_scheduler_t` in Phase 1
tests uses the same field names and types, so the predicate wrapper
bodies are mechanically identical — only the container struct pointer
differs.
These fields are initialized to 0 in the core's init function and
set/cleared at the appropriate session-loop transitions (see
`src/scheduler/src/core_session.c`, referenced via `core->session_active`,
`core->eval_in_progress`, `core->staging_merge_complete`). The predicate
wrappers read them via `atomic_load_explicit(..., memory_order_acquire)`
to ensure visibility from the allocator thread's writes. The
`fake_scheduler_t` in Phase 1 tests uses the same field names and types,
so the predicate wrapper bodies are mechanically identical — only the
container struct pointer differs.
**API shape:** mark and sweep are two separate exported functions, not one
combined pass. The scheduler (caller) owns the mark phase in full —
including the `resreq->tag` dispatch over live jobs — and only then invokes
the sweep:
```
gc_clear_marks(void)
Iterate sel_node_table.nodes (zarray_t) over slots 0..caps-1: skip NULL
slots (vacated by zarray_del from a prior GC sweep); clear gc_mark on
every live node.
gc_mark_select_tree(xce_select_node_t* node)
Recursive DFS from `node`, setting gc_mark on every reachable node.
Early-returns on a NULL node or a node whose gc_mark is already set, so
shared sub-nodes are marked exactly once even though multiple jobs reach
them. The **caller** (scheduler) invokes this once per live job's
select root(s), performing the resreq->tag dispatch itself:
- XCE_RESREQ_PRIMARY: gc_mark_select_tree(resreq->u.primary->select->root).
- XCE_RESREQ_COMPOUND: iterate resreq->u.compound->primaries and call
gc_mark_select_tree on each primary's select->root.
- XCE_RESREQ_ALTER: iterate resreq->u.alter->compounds, then each
compound's primaries, then call gc_mark_select_tree on each primary's
select->root.
(xce_sched_job_t has xce_resreq_t* resreq; xce_job_t only has char* resreq
and cannot access the parsed AST.)
gc_sweep_select_nodes(const xce_sched_job_t** live_jobs, size_t njobs)
1. Mark pass:
a. Iterate sel_node_table.nodes (zarray_t) over slots 0..caps-1:
skip NULL slots (vacated by zarray_del from a prior GC sweep);
clear gc_mark on every live node.
b. For each live job in live_jobs[0..njobs-1]: dispatch on
job->resreq->tag and walk all reachable select roots, setting
gc_mark on every reachable node. (Shared sub-nodes are marked
once — gc_mark is boolean.)
- XCE_RESREQ_PRIMARY: mark from resreq->u.primary->select->root.
- XCE_RESREQ_COMPOUND: iterate resreq->u.compound->primaries and
mark each primary's select->root.
- XCE_RESREQ_ALTER: iterate resreq->u.alter->compounds, then each
compound's primaries, then each primary's select->root.
(xce_sched_job_t has xce_resreq_t* resreq; xce_job_t only has char* resreq
and cannot access the parsed AST.)
2. Sweep pass:
a. Iterate sel_node_table.nodes (zarray_t) over slots 0..caps-1:
skip NULL slots; collect (slot_idx, node*) pairs for unmarked nodes.
**Note:** `slot_idx` here is the loop index `i` from the zarray
iteration — it is NOT stored in `xce_select_node_t` (the struct has no
`slot_idx` field). The GC sweep obtains the correct `zarray_del` argument
directly from the forward-iteration index, not from any per-node field.
b. For each unmarked node (iterate collected pairs):
- Reconstruct the intern key: `sel_node_key_t key = sel_node_to_key(node)`
(see §12 Appendix). The commutativity sort must be applied during
key reconstruction, identical to the sort applied in `makeSelectNode`
step 1, otherwise the reconstructed key hashes to a different bucket
and `key_del` silently no-ops, leaking the kvtab entry.
- Remove its key from sel_node_table.table via key_del.
(sel_node_table is initialized with datakiller=NULL so key_del
removes only the hash table entry — the node value is NOT freed by
key_del; see discussion below on datakiller.)
- If node->type == SEL_STR: call `dereference(node->u.strval)` to
release the xstring refcount acquired by `makeString()`.
All SEL_STR nodes are created via `makeString()`, so strval is
always an xstring-interned pointer — `free()` is incorrect here.
- free(node).
- zarray_del(&sel_node_table.nodes, slot_idx) — NULLs the slot and
decrements count. Does NOT swap-remove; remaining slot indices are
stable. NULL-slot skipping in step 1a handles future iterations.
Sweep-only — does **not** mark. The mark phase above
(gc_clear_marks + gc_mark_select_tree per live node) must already have
been run by the caller before this function is invoked; `live_jobs` and
`njobs` are retained only for API-compatibility and are unused inside
the function body (`(void)live_jobs; (void)njobs;`). Every production
call site therefore invokes `gc_sweep_select_nodes(NULL, 0)` after
completing its own mark phase.
a. Iterate sel_node_table.nodes (zarray_t) over slots 0..caps-1:
skip NULL slots; collect (slot_idx, node*) pairs for unmarked nodes.
**Note:** `slot_idx` here is the loop index `i` from the zarray
iteration — it is NOT stored in `xce_select_node_t` (the struct has no
`slot_idx` field). The GC sweep obtains the correct `zarray_del` argument
directly from the forward-iteration index, not from any per-node field.
b. For each unmarked node (iterate collected pairs):
- Reconstruct the intern key: `sel_node_key_t key = sel_node_to_key(node)`
(see §12 Appendix). The commutativity sort must be applied during
key reconstruction, identical to the sort applied in `makeSelectNode`
step 1, otherwise the reconstructed key hashes to a different bucket
and `key_del` silently no-ops, leaking the kvtab entry.
- Remove its key from sel_node_table.table via key_del.
(sel_node_table is initialized with datakiller=NULL so key_del
removes only the hash table entry — the node value is NOT freed by
key_del; see discussion below on datakiller.)
- If node->type == SEL_STR: call `dereference(node->u.strval)` to
release the xstring refcount acquired by `makeString()`.
All SEL_STR nodes are created via `makeString()`, so strval is
always an xstring-interned pointer — `free()` is incorrect here.
- free(node).
- zarray_del(&sel_node_table.nodes, slot_idx) — NULLs the slot and
decrements count. Does NOT swap-remove; remaining slot indices are
stable. NULL-slot skipping in step 1a handles future iterations.
```
The `gc_mark_tree` helper does a recursive post-order mark:
**Protocol summary:** `gc_clear_marks()` → `gc_mark_select_tree(root)` per
live node → `gc_sweep_select_nodes(NULL, 0)` → `recompact_node_ids()`. Both
real call sites (`src/scheduler/src/scheduler.c` and
`src/scheduler/src/core_session.c`) follow this exact sequence, with an
`atomic_thread_fence(memory_order_seq_cst)` between the mark phase and the
sweep call to make mark writes visible before the sweep reads them.
The `gc_mark_select_tree` helper does a recursive post-order mark:
```c
/* Mark all nodes reachable from root (recursive DFS).
@ -1166,8 +1203,14 @@ The `gc_mark_tree` helper does a recursive post-order mark:
* Recursion depth: bounded by the post-flatten AST depth, which is at most
* SELECT_MAX_DEPTH - 1 levels (the parse-time nesting limit). Any increase
* to SELECT_MAX_DEPTH must be reviewed against stack-frame size to avoid
* stack overflow. Cycles are impossible — interned nodes form a DAG. */
static void gc_mark_tree(xce_select_node_t* node)
* stack overflow. Cycles are impossible — interned nodes form a DAG.
*
* As implemented, this is a non-static function exported from
* `xce_select.c` and declared in the public `resreq.h` header — not a
* `static` helper local to `resreq.c` — because the mark phase, including
* the resreq->tag dispatch below, is the caller's (scheduler's)
* responsibility. See §5.5. */
void gc_mark_select_tree(xce_select_node_t* node)
{
if (node == NULL || node->gc_mark) return; /* NULL root or already marked */
node->gc_mark = 1;
@ -1178,12 +1221,12 @@ static void gc_mark_tree(xce_select_node_t* node)
break;
/* unary */
case SEL_NOT:
gc_mark_tree(node->u.operand);
gc_mark_select_tree(node->u.operand);
break;
/* binary */
default: /* SEL_AND, SEL_OR, SEL_GT, SEL_LT, SEL_GE, SEL_LE, SEL_EQ, SEL_NE */
gc_mark_tree(node->u.binop.left);
gc_mark_tree(node->u.binop.right);
gc_mark_select_tree(node->u.binop.left);
gc_mark_select_tree(node->u.binop.right);
break;
}
}
@ -1200,7 +1243,7 @@ xlist_t* list = &resreq->u.compound->primaries;
for (int i = 0; i < xlist_length(list); i++) {
xce_primary_resreq_t* p = (xce_primary_resreq_t*)xlist_get(list, i);
if (p && p->select && p->select->root)
gc_mark_tree(p->select->root);
gc_mark_select_tree(p->select->root);
}
```
@ -1904,9 +1947,9 @@ eval) or data races (concurrent makeSelectNode + eval without the intern
table lock). Phase 6 must enforce this with a runtime thread-ID check
in `eval_select` and `gc_sweep_select_nodes`.
**GC single-thread guarantee:** `gc_mark_tree` writes the non-atomic
`gc_mark` field on interned nodes. This is safe because `gc_sweep_select_nodes`
is called exclusively from the allocator thread (a single-threaded context
**GC single-thread guarantee:** `gc_mark_select_tree` writes the non-atomic
`gc_mark` field on interned nodes. This is safe because the mark phase and
`gc_sweep_select_nodes` are called exclusively from the allocator thread (a single-threaded context
— see §8.5.3 pre-session lifecycle). All `gc_mark` writes from a single
GC sweep are sequential; no second thread accesses `gc_mark` concurrently.
If the single-thread guarantee is ever relaxed, `gc_mark` must be made
@ -2200,15 +2243,19 @@ memory savings shown in §8.5.5.
│ atomic_thread_fence(memory_order_seq_cst) must be inserted after │
│ all `primary->select->root = NULL` writes and before gc_sweep. │
│ Without the fence, the CPU or compiler may reorder the NULL store │
│ after the GC mark pass begins, causing gc_mark_tree to observe a │
│ stale non-NULL root and retain a node that should be swept. │
│ after the GC mark pass begins, causing gc_mark_select_tree to │
│ observe a stale non-NULL root and retain a node that should be │
│ swept. │
│ In C11: atomic_thread_fence(memory_order_seq_cst); │
│ Equivalent: make `select->root` an _Atomic pointer and use │
│ atomic_store_explicit(..., memory_order_release) for the unlink. │
│ 4. Run gc_sweep_select_nodes(live_jobs, njobs) — mark pass visits │
│ all live jobs including newly arrived ones, so nodes referenced │
│ only by incoming jobs are correctly retained. │
│ 5. Run recompact_node_ids() — reassign dense 0..N-1 IDs │
│ 4. gc_clear_marks(); gc_mark_select_tree(root) per live job's select │
│ root(s) — mark pass visits all live jobs including newly arrived │
│ ones, so nodes referenced only by incoming jobs are correctly │
│ retained. (Caller-driven mark phase — see §5.5.) │
│ 5. Run gc_sweep_select_nodes(NULL, 0) — sweep only; mark already │
│ done in step 4 above (live_jobs/njobs args unused — see §5.5). │
│ 6. Run recompact_node_ids() — reassign dense 0..N-1 IDs │
│ After recompact: current_node_count = sel_node_next_id │
│ = count of live interned nodes with dense IDs 0..N-1. │
└───────────────────────────────────────────────────────────────────────┘
@ -2542,7 +2589,8 @@ are made.
scheduler wiring. In Phase 1, these predicates are backed by a
**test-only `fake_scheduler_t` struct** (declared in `test_select.c`)
with the same three `_Atomic int` fields that Phase 6 will add to the
real `xce_scheduler_t` (see §5.5 "Predicate lifecycle" and Phase 6
real scheduler core struct (as implemented: `struct xce_sched_core_`
in `core.h` — see §5.5 "Predicate lifecycle" and Phase 6
"Add GC predicate fields"). The Phase 1 stub functions wrap the fake
struct; Phase 6 replaces them with wrappers around the real struct.
No Phase 1 stub code leaks into the production implementation — the
@ -2567,14 +2615,20 @@ The grammar is the test oracle for all parse-level assertions.*
- Fill `struct xce_resreq_select_` with `xce_select_node_t* root`.
- Declare `makeSelectNode`, `eval_select`, `eval_select_inner`,
`select_eval_ctx_init`, `select_eval_ctx_reset`, `select_eval_ctx_fini`,
`sel_node_table_init`, `sel_node_table_fini`, `gc_sweep_select_nodes`,
`recompact_node_ids`.
(`gc_mark_tree` is a `static` helper in `resreq.c` — §5.5 — and must
NOT be declared in the public header.)
`sel_node_table_init`, `sel_node_table_fini`, `gc_clear_marks`,
`gc_mark_select_tree`, `gc_sweep_select_nodes`, `recompact_node_ids`.
(As implemented, `gc_clear_marks` and `gc_mark_select_tree` ARE
declared in the public header and exported from `xce_select.c` —
not `static` helpers local to `resreq.c` — because the mark phase,
including the resreq->tag dispatch, is the caller's (scheduler's)
responsibility; see §5.5.)
- `gc_sweep_select_nodes` signature:
`void gc_sweep_select_nodes(const xce_sched_job_t** live_jobs, size_t njobs);`
(`xce_sched_job_t` has `xce_resreq_t* resreq`; `xce_job_t` has only
`char* resreq` and cannot reach the parsed AST.)
`char* resreq` and cannot reach the parsed AST. As implemented, the
`live_jobs`/`njobs` parameters are unused inside the function — every
call site passes `NULL, 0` after running the mark phase separately
via `gc_clear_marks`/`gc_mark_select_tree`; see §5.5.)
Caller: scheduler pre-session, after staging merge. Called after all job
cleanup (select root unlinks) and before `recompact_node_ids`.
Precondition: no `eval_select` call is in progress (GC idle invariant, §5.5).
@ -2935,8 +2989,11 @@ allocations — but these must be found and updated.
`host_get_string` accessors.
**4f. Mark-and-sweep GC and recompaction (§5.5/§8.5.4)**
- `gc_sweep_select_nodes()` with resreq->tag dispatch for COMPOUND/ALTER.
- `gc_mark_tree()` helper.
- `gc_clear_marks()` and `gc_mark_select_tree()` — mark phase; the
caller performs the resreq->tag dispatch for COMPOUND/ALTER and
invokes `gc_mark_select_tree()` per reachable select root.
- `gc_sweep_select_nodes()` — sweep-only; the caller must run the mark
phase above first (see §5.5).
- `recompact_node_ids()` — reassign dense 0..N-1 IDs after GC (§8.5.4).
#### Phase 5: Build wiring
@ -3173,27 +3230,34 @@ wires the select evaluator into the scheduler's existing session loop:
**GC phase (§8.5.3 "Staging + GC phase"):**
1. Merge new jobs from staging buffer, parse their resreq strings
(calling `makeSelectNode`), and unlink completed jobs.
2. `gc_sweep_select_nodes(live_jobs, njobs)` — reclaim unreachable nodes.
Running GC after the staging merge ensures nodes referenced only by
newly arrived jobs are marked live and retained.
2. `gc_clear_marks()` then `gc_mark_select_tree(root)` per live job's
select root(s) (dispatching on `resreq->tag`), then
`gc_sweep_select_nodes(NULL, 0)` — reclaim unreachable nodes. As
implemented, marking is the caller's responsibility and is a
separate step from the sweep call — see §5.5. Running GC after the
staging merge ensures nodes referenced only by newly arrived jobs
are marked live and retained.
3. `recompact_node_ids()` — reassign dense 0..N-1 IDs (§8.5.4).
**Pre-session capacity check (§8.5.3 "Pre-session capacity check"):**
4. Check whether eval context arrays need to grow or shrink to
accommodate the new `max_node_id` (§8.5.3 step 1).
- **Add GC predicate fields to `xce_scheduler_t`** in
`src/scheduler/include/scheduler.h`: add `_Atomic int session_active`,
`_Atomic int eval_in_progress`, and `_Atomic int staging_merge_complete`
to the struct. These back the predicate functions below; the struct
declaration must be updated before the functions can reference them.
Use `_Atomic int` (not `volatile int`) to match the `_Atomic uint32_t
sel_node_next_id` pattern (§5.3) and to provide correct atomicity on ARM
where `volatile` alone does not guarantee atomic reads/writes.
- **Add GC predicate fields to the scheduler's core struct**: add
`_Atomic int session_active`, `_Atomic int eval_in_progress`, and
`_Atomic int staging_merge_complete`. These back the predicate
functions below; the struct declaration must be updated before the
functions can reference them. Use `_Atomic int` (not `volatile int`)
to match the `_Atomic uint32_t sel_node_next_id` pattern (§5.3) and to
provide correct atomicity on ARM where `volatile` alone does not
guarantee atomic reads/writes. **As implemented,** these fields live
in `struct xce_sched_core_` (`src/scheduler/include/core.h`), not
`struct xce_scheduler_` (`scheduler.h`) — see §5.5.
- **GC predicate implementation:** The `sched_session_active`,
`sched_eval_in_progress`, and `staging_merge_complete` predicates
(§5.5) wrap the new fields above added to `xce_scheduler_t` in Phase 6.
(§5.5) wrap the new fields above added to the scheduler's core struct
in Phase 6.
Lifecycle of each flag (all use `_Atomic int` — use
`atomic_store_explicit(..., memory_order_release)` for writes and
@ -3238,11 +3302,16 @@ wires the select evaluator into the scheduler's existing session loop:
requires runtime guards for safety-critical invariants — `assert()` is
compiled out with `NDEBUG` and must not be relied upon here.
- **live_jobs materialization:** `gc_sweep_select_nodes` takes a flat
`const xce_sched_job_t**` array, but the scheduler stores jobs in
`sched->jobs` as a `var`/kvmgr keyed by jobid. Phase 6 must
materialize this array using a plain heap-allocated C array to avoid
the fragile cast assumption of `(const xce_sched_job_t**)zarray_t.data`:
- **live_jobs materialization:** The mark phase (`gc_clear_marks()` then
`gc_mark_select_tree()` per live job's select root(s), dispatching on
`resreq->tag` — see §5.5) needs a flat list of live jobs, but the
scheduler stores jobs in `sched->jobs` as a `var`/kvmgr keyed by jobid.
Phase 6 must materialize this array using a plain heap-allocated C
array to avoid the fragile cast assumption of
`(const xce_sched_job_t**)zarray_t.data`. `gc_sweep_select_nodes`
itself is called afterward as `gc_sweep_select_nodes(NULL, 0)` — it
does not take the array (see §5.5); the array only feeds the mark
phase:
```c
size_t live_cap = 64, n = 0;
@ -3263,7 +3332,12 @@ wires the select evaluator into the scheduler's existing session loop:
}
live_arr[n++] = job;
}
gc_sweep_select_nodes(live_arr, n);
gc_clear_marks();
for (size_t i = 0; i < n; i++) {
/* dispatch on live_arr[i]->resreq->tag and call gc_mark_select_tree()
* on each reachable select root — see §5.5 */
}
gc_sweep_select_nodes(NULL, 0);
free(live_arr);
```

12
src/docs/xce.md
View file

@ -9,11 +9,13 @@ available:
- **scheduler** — start xce scheduler daemon
- **apiserver** — start xce REST API server
- **shell** — interactive bash shell for troubleshooting
- **xcecli** — CLI tool for cluster interaction (job submit, query, admin)
If you run xce without any parameter or with a parameter not listed above, it
will show this help screen.
Note: `xcecli` is not a sub-command of `xce` — it is a separate standalone
Go binary invoked directly. See the [xcecli](#xcecli) section below.
## controller
Start the xce controller daemon with the given parameters after the
@ -21,7 +23,7 @@ Start the xce controller daemon with the given parameters after the
Example:
```
xce controller --config /etc/xce/controller.yaml
xce controller --workdir /var/lib/xce/controller --myname ctrl01 --peers ctrl01:9990,ctrl02:9990
```
## scheduler
@ -45,9 +47,9 @@ authentication and RBAC authorization.
The xce CLI (`src/frontend/xcecli/`) provides command-line access to the
cluster. Common operations:
- `xcecli job submit <jobspec>` — submit a job
- `xcecli job list` — list jobs
- `xcecli queue status` — show queue state
- `xcecli jobs submit <jobspec>` — submit a job
- `xcecli jobs list` — list jobs
- `xcecli queues list` — show queue state
Built as a standalone Go binary; communicates with the API server over
its REST interface.

15
src/lib/CLAUDE.md
View file

@ -1,29 +1,28 @@
# XCE Shared Library (libxce)
## Overview
Shared C library providing common utilities, data structures, and IPC primitives used by the controller, worker, and scheduler daemons.
Shared C library providing common utilities, data structures, and IPC primitives used by the controller, worker, and scheduler daemons, plus the Go apiserver (via CGO, for IPC with the controller).
## Build
```bash
# From workspace root
nix develop
meson build && ninja -C build
/xce-build lib # debug + ASAN + clean build + runs tests
```
For full build system details, see `.kiro/steering/build-deployment.md`.
## Architecture
- Produces `libxce.a` (static) and/or `libxce.so` (shared)
- Produces `libxce.a` (static only — the `shared_library('xce', ...)` target in `meson.build` is currently commented out; no `.so` is built)
- Linked by all C system components: controller, worker, scheduler
- Public headers exposed via `src/include/xce/`
- Public headers consumed by all components live flat in `src/include/` (e.g. `client.h`, `netio.h`, `request.h`, `vtaskmgr.h`, `persist.h`); only a handful of headers (`client_failover.h`, `cluster.h`, `cluster_def.h`, `errnox.h`, `types.h`) live under `src/include/xce/`
- Test suite in `src/lib/testsuite/`
## Conventions
- C standard: gnu17
- Format with `clang-format` (config at repo root if present)
- All public API headers go in `src/include/xce/` — not in `src/lib/`
- Public API headers go in `src/include/` (most headers) or `src/include/xce/` (a small legacy subset) — not in `src/lib/`
- Implementation files stay in `src/lib/`
## Key Files
- `src/include/xce/` — Public header files consumed by all components
- `src/include/` — Public header files consumed by all components (`src/include/xce/` holds a small legacy subset)
- `src/lib/testsuite/` — Library unit tests
## See Also

6
src/scheduler/CLAUDE.md
View file

@ -10,10 +10,10 @@ C-based job scheduling engine with a plugin architecture. Determines job-to-node
```
## Architecture
- **Core** (`core.c`/`core_events.c`/`core_session.c`/`core_clock.c`) — passive, event-driven `xce_sched_core_t` state machine. Zero controller I/O (no `netio`/`xceapi`/`client`/`delay`/`idgen` — enforced by `test_core_isolation.c`); `core_clock.c` is the one sanctioned exception (`clock_gettime`, virtualized two-base+speed clock formula, `xce_core_now()`). Picker→allocator handoff is a strict `jbuf` cbuf capacity-1 rendezvous — this is the project's determinism invariant, not a tuning knob; never raise it without re-deriving Req 5 AC 7 in `.kiro/specs/scheduler-core/requirements.md`.
- **Core** (`core.c`/`core_events.c`/`core_session.c`/`core_clock.c`) — passive, event-driven `xce_sched_core_t` state machine. Zero controller I/O (no `netio`/`xceapi`/`client`/`delay`/`idgen`); `core_clock.c` is the one sanctioned exception (`clock_gettime`, virtualized two-base+speed clock formula, `xce_core_now()`). `test_core_isolation.c` mechanically enforces this only for `core.c`/`core.h` (source-text scan + `nm(1)` undefined-symbol check on the isolated `sched_core_isolation_lib` archive, meson.build) — `core_events.c`, `core_session.c`, and `core_clock.c` are not covered by that test and must be kept isolation-clean by review. Picker→allocator handoff is a strict `jbuf` cbuf capacity-1 rendezvous — this is the project's determinism invariant, not a tuning knob; never raise it without re-deriving Req 5 AC 7 in `.kiro/specs/scheduler-core/requirements.md`.
- **Picker plugin interface** (`plugin.h`'s `xce_job_picker_iface_t`, `plugins/builtin_fcfs.c`) — job-selection strategies are pluggable; `scheduler.c`'s `sched_job_picker()` branches on whether a plugin still uses the legacy `job_elect_fn`-only shape (`plugins/fcfs.c`, kept as-is for a compat-wrapper exercise, not the live default) vs the native hook interface (`builtin_fcfs`, the registered default). New scheduling policies → add as a picker plugin under `src/plugins/`, do not modify the core loop.
- **Drivers** — `driver.c` (normal mode: capnp↔core event translation, decision publishing, deferred admin-request handling) and `sim.c`/`sim_loop.c`/`sim_record.c`/`sim_verify.c` (simulation mode: `.replay` file loading, discrete virtual clock, `--sim-verify`/`--sim-alt-events`/`--sim-output`). Both drive the identical core session/decision-callback protocol — one core, two consumers.
- **Replay/recording** — `recording.c`/`recording_events.c`/`recording_session.c` (live capture) and `snapshot.c`/`snapshot_wire.c`/`snapshot_load.c` (state save/restore) read/write the `.replay` capnp binary stream (`src/schema/controller.capnp`); controlled via `OPCode_schedAdmin` (`src/controller/cluster.c`) and `xceadmin scheduler record start|stop|status` / `scheduler dump <FILE>`.
- **Replay/recording** — `recording.c`/`recording_events.c`/`recording_session.c` (live capture) and `sim.c`/`sim_loop.c` (simulation-side decode) do the actual capnp encode/decode of the `.replay` binary stream (`src/schema/controller.capnp`); `snapshot.c`/`snapshot_wire.c`/`snapshot_load.c` only populate/consume the plain in-memory `sched_snapshot_data_t` struct that `recording.c`'s `write_snapshot_entry()` then serializes — no capnp or file I/O lives in the snapshot files themselves. Controlled via `OPCode_schedAdmin` (`src/controller/cluster.c`) and `xceadmin scheduler record start|stop|status` / `scheduler dump <FILE>`.
- **Interactive REPL** (`repl.c`/`repl_break.c`/`repl_info.c`/`repl_params.c`, `--interactive`) — single-step/breakpoint/state-inspection/snapshot-capture control over the simulation driver; drives the exact same `sched_sim_driver_run()` as batch `--sim`, gated only by `sim->interactive`, so REPL and batch runs are decision-equivalent by construction (`test_repl_equivalence.c`).
Links against `libxce` from `src/lib/`. Full design/requirements: `.kiro/specs/scheduler-core/`.
@ -23,7 +23,7 @@ Links against `libxce` from `src/lib/`. Full design/requirements: `.kiro/specs/s
- Format with `clang-format`
- New scheduling policies → add as a plugin under `src/plugins/`, do not modify the core engine
- Shared types defined in `src/include/xce/` (libxce)
- Files stay under the repo's 800-line cap — several core/sim/repl/snapshot/recording modules are deliberately split across multiple `*_events.c`/`*_session.c`/`*_wire.c`-style files for this reason; keep new logic in the file matching its concern rather than growing one file past the cap.
- Files stay under the repo's 800-line cap — several core/sim/repl/snapshot/recording modules are deliberately split across multiple `*_events.c`/`*_session.c`/`*_wire.c`-style files for this reason; keep new logic in the file matching its concern rather than growing one file past the cap. Not yet true project-wide: `scheduler.c` (1382 lines) and `plugins/builtin_fcfs.c` (1026 lines) are pre-existing exceptions over the cap with no split or exemption note — do not use them as precedent for new files, and prefer splitting them opportunistically rather than growing them further.
## See Also
- Shared library: `src/lib/CLAUDE.md`

10
src/worker/CLAUDE.md
View file

@ -5,15 +5,13 @@ C daemon running on each compute node. Manages job execution, hardware monitorin
## Build
```bash
# From workspace root
nix develop
meson build && ninja -C build
/xce-build worker # lib + worker daemon (see build-deployment.md)
```
## Architecture
- `src/xcecd.c` — Main daemon entry point
- `src/hardware_monitor.c` — CPU/memory/GPU resource monitoring
- `src/process_manager.c` — Job process lifecycle (spawn, track, reap)
- `src/main.c` — Main daemon entry point (`int main`); links against `xcecd_srcs` from `src/xcecd.c` and friends
- `src/hardware_monitor.c` — CPU/memory resource monitoring (no GPU)
- `src/process_worker_manager.c` + `src/uv_proc.c` — Job process lifecycle (fork `xcecd_worker`, spawn/exec the job command); `src/process_manager.c` is unused dead code for job spawn (its `process_manager_start_process` is never called) but is still wired up for generic create/update/destroy/reap in `xcecd.c`
Links against `libxce` from `src/lib/`.