claude skills

.kiro/specs/ha-config-bootstrap/requirements.md

@@ -0,0 +1,151 @@
+# Requirements: HA Config Bootstrap & Raft-Style Quorum Election
+
+## Introduction
+
+The current LSF-style priority-based controller election has three critical safety gaps: (1) no guarantee of config freshness on the elected primary, (2) false self-elections during concurrent startup when the peer listener port is not yet open, and (3) no bootstrap path for a standby promoted during genuine failover whose local config is missing. This feature replaces priority-based election with Raft-style quorum voting, adds majority-commit WAL and config replication for full CP durability, establishes persistent standby-to-primary control channels, and hardens the controller startup sequence. The system shall maintain CP semantics: consistency over availability, with zero data loss on failover.
+
+## Boundary Context
+
+- **In scope**: quorum-based controller election, pre-election config bootstrap for unconfigured nodes, listener binding before election, startup pre-flight validation (peer count, certs, vcpus), majority-commit WAL replication with entry-type differentiated durability, majority-commit config push, persistent standby-to-primary control channels with catch-up synchronization, client-side transparent retry for transient controller errors, election term and voted_for durability across restarts, observable election and commit events via structured logs
+- **Out of scope**: Raft snapshot protocol (existing COW is orthogonal), joint consensus for cluster membership changes (candidate list is static at startup from `--peers`), mTLS between controllers (trusted-LAN model with server-side TLS is sufficient), config file format or storage layout changes
+- **Adjacent expectations**: the existing `do_one_config_fetch()` stub must be completed for config bootstrap to function in the deferred-fetch path; static controller peer membership (no dynamic join/leave) is a hard constraint; single-node mode (quorum=1) must remain fully operational without behavioral changes
+
+## Requirements
+
+### Requirement 1: Listener and Connection Framework Before Election
+
+**Objective:** As a cluster operator, I want peer listener sockets to be open before election begins, so that concurrent-startup nodes can reach each other's vote handlers rather than falsely concluding peers are absent.
+
+#### Acceptance Criteria
+1. When a controller starts, the system shall bind and listen on its peer port before initiating any election probe.
+2. When a controller starts, the system shall initialize the I/O manager and virtual task manager before initiating any election probe, so that incoming vote requests are handled by the normal dispatch loop.
+3. When the controller is in the startup-initializing or electing state, the system shall reject non-vote, non-config-probe requests with a not-ready error response.
+4. When the controller is in standby state, the system shall reject primary-only operations with a standby error response that includes a redirect to the known primary host and port.
+
+---
+
+### Requirement 2: Quorum-Based Controller Election
+
+**Objective:** As a cluster operator, I want the controller election to require a strict majority of peer votes, so that a minority partition cannot elect a primary and split-brain is prevented.
+
+#### Acceptance Criteria
+1. When a controller enters the electing state, the system shall send vote requests to all peer controllers and require acknowledgments from a strict majority (greater than or equal to half the cluster size rounded up) before becoming primary.
+2. When a controller receives a vote request from a candidate with a lower term, the system shall deny the vote.
+3. When a controller receives a vote request from a candidate with a higher term, the system shall update its local term and clear any prior vote record for the previous term.
+4. When a controller receives a vote request from a candidate whose config tag or WAL log sequence number is less fresh than its own, the system shall deny the vote.
+5. When a peer controller is in catch-up synchronization and has not yet caught up to the primary's committed state, the system shall deny its vote in any election.
+6. When a controller has already granted a vote to one candidate in a given term, the system shall deny any subsequent vote request from a different candidate in the same term.
+7. When a candidate wins the vote count but any responding peer has a higher config tag or equal config tag with a higher WAL log sequence number, the system shall yield to that fresher peer and enter standby state.
+8. When a candidate with no local config (config tag zero) determines that any peer has a config with a non-zero tag, the system shall refuse to become primary and exit with a fatal error.
+9. When the cluster is a single node, the system shall self-elect as primary immediately with no quorum requirement.
+10. When a split vote occurs where no candidate achieves a quorum, the system shall retry the election once with a randomized delay and incremented term.
+11. If the second election attempt also fails to achieve a quorum, the system shall enter standby state.
+12. When every responding peer has an identical config tag and WAL log sequence number as the candidate, the system shall use randomized pre-election jitter as the tiebreaker without considering priority values.
+
+---
+
+### Requirement 3: Config Bootstrap for Unconfigured Nodes
+
+**Objective:** As a cluster operator, I want a controller node that starts with no local configuration to automatically discover and fetch configuration from the proven election winner after the election resolves, so that fresh nodes can join an existing cluster without manual intervention.
+
+#### Acceptance Criteria
+1. When a controller starts with no local config (config tag zero), the system shall probe all peer controllers for their config tag before declaring candidacy.
+2. When a controller with no local config detects that at least one peer has a non-zero config tag, the system shall participate in the election as a voter only and defer the config fetch until the election winner is known.
+3. When a controller with no local config and all reachable peers also have no config, the system shall proceed to election as a normal candidate with config tag zero.
+4. When a controller with no local config cannot reach any peer, the system shall retry with exponential backoff (starting at 500 milliseconds, doubling up to 10 seconds) until a peer responds or the process is killed by an administrator.
+5. When the election resolves and a deferred config fetch is pending, the system shall fetch the full configuration from the proven election winner.
+
+---
+
+### Requirement 4: Startup Validation and Safety Checks
+
+**Objective:** As a cluster operator, I want the controller to validate its startup prerequisites before initialization, so that misconfigured deployments are detected immediately with clear, actionable error messages.
+
+#### Acceptance Criteria
+1. When the `--peers` list contains an even number greater than 2, the system shall reject the configuration at startup with an error message explaining that an odd number of controllers is required for majority formation.
+2. When the TLS certificate or key file is missing or unreadable at the expected path, the system shall reject startup with an error message that includes the full file path.
+3. When the `--vcpus` value is less than 1, the system shall reject startup with an error message.
+4. When the cluster config file contains a `vcpus` key, the system shall log a deprecation warning directing operators to use the `--vcpus` CLI parameter instead.
+
+---
+
+### Requirement 5: Majority-Commit WAL Replication
+
+**Objective:** As a cluster operator, I want WAL entries for critical state changes to be committed only after a majority of controllers have durably written them, so that zero data loss is guaranteed on failover.
+
+#### Acceptance Criteria
+1. When the primary writes a job submission, job state transition, or queue change WAL entry, the system shall not acknowledge the write to the caller until a majority of controllers (including the primary itself) have confirmed durable write of that entry.
+2. When the primary writes a host state or checkpoint reference WAL entry, the system shall acknowledge the write without waiting for standby confirmation.
+3. When a standby controller receives a WAL entry from the primary, the system shall write it to durable storage and reply with an acknowledgment carrying the written log sequence number.
+4. When a standby controller falls more than the configurable maximum pending acknowledgments behind the primary, the system shall remove that standby from the active replication set and trigger catch-up reconnection.
+5. If the primary cannot obtain majority acknowledgment within the configured commit timeout, the system shall return a timeout error to the caller.
+6. When a standby controller is promoted to primary, the system shall truncate any WAL entries above the primary's last known committed log sequence number before replaying the WAL.
+7. When a standby controller reconnects to a primary via catch-up, the system shall truncate any uncommitted WAL entries above the primary's committed log sequence number before requesting WAL replay.
+8. When the cluster is a single node, the system shall consider the local write alone as satisfying the majority requirement.
+
+---
+
+### Requirement 6: Persistent Standby-to-Primary Control Channel
+
+**Objective:** As a cluster operator, I want standby controllers to maintain a persistent connection to the primary, so that failover detection is sub-second and WAL replication avoids per-entry connection overhead.
+
+#### Acceptance Criteria
+1. When a standby controller enters standby state after election, the system shall establish and maintain a persistent TLS connection to the elected primary.
+2. When the persistent control channel disconnects, the system shall detect the failure immediately and trigger a new election.
+3. When heartbeat messages are received on the persistent control channel, the system shall reset the heartbeat timeout and defer election.
+4. When no heartbeat is received within the election timeout plus randomized jitter on the persistent control channel, the system shall trigger a new election.
+5. When a standby with an outdated config tag or applied log sequence number connects to the primary, the system shall synchronize all missing state (config snapshot and WAL entries) before admitting the standby to the active replication set.
+6. When a standby's applied log sequence number is older than the primary's oldest retained WAL segment, the system shall transfer a full state snapshot before streaming WAL entries.
+7. When a standby connects with a config tag that differs from the primary's, the system shall push the primary's current configuration after catch-up synchronization completes.
+8. When a standby wins an election and becomes primary, the system shall close its outbound connection to the old primary and begin accepting inbound control channel connections from other standbys.
+
+---
+
+### Requirement 7: Client-Side Transparent Retry for Transient Controller Errors
+
+**Objective:** As a daemon developer, I want libxce to transparently retry requests when the controller returns transient errors, so that worker, scheduler, and API server code does not need ad-hoc retry loops.
+
+#### Acceptance Criteria
+1. When libxce receives a not-ready error from a controller, the system shall retry the request with exponential backoff against the same endpoint.
+2. When libxce receives a standby error that includes a redirect to the current primary, the system shall reconnect to the redirected host and port and retry the request.
+3. When libxce receives a TCP-level connection refused or timeout error, the system shall reconnect and retry the request.
+4. When libxce receives a non-retriable error (authentication failure, invalid request, permission denied, not found), the system shall surface the error to the caller immediately without retry.
+5. The system shall provide configurable retry parameters (maximum retries, initial delay, maximum delay, backoff multiplier) with safe production defaults.
+6. When multiple clients retry simultaneously after a failover, the system shall stagger their retry attempts through exponential backoff to prevent thundering herd.
+
+---
+
+### Requirement 8: Majority-Commit Config Replication
+
+**Objective:** As a cluster operator, I want configuration changes pushed to the cluster to be committed only after a majority of controllers have durably written them, so that operator changes are not silently lost on primary failover.
+
+#### Acceptance Criteria
+1. When the primary pushes a configuration upgrade, the system shall not acknowledge the change as committed until a majority of controllers (including the primary itself) have confirmed durable write.
+2. When a standby controller receives a configuration upgrade from the primary, the system shall write the config blob to durable storage and reply with an acknowledgment.
+3. If the primary cannot obtain majority acknowledgment for a configuration upgrade within the configured commit timeout, the system shall return a timeout error to the caller.
+4. When a standby is promoted to primary, the system shall serve only configurations up to the last known committed config tag received via heartbeat from the previous primary.
+5. When the cluster is a single node, the system shall consider the local write alone as satisfying the majority requirement for config upgrades.
+
+---
+
+### Requirement 9: Election State Durability Across Restarts
+
+**Objective:** As a cluster operator, I want the controller's election term and vote record to survive process restarts, so that a restarting node cannot double-vote in a term it already participated in.
+
+#### Acceptance Criteria
+1. When a controller's term is incremented or its vote is granted to a candidate, the system shall durably persist the current term and voted-for candidate name before responding to the vote request.
+2. When a controller starts up, the system shall restore the most recently persisted term and voted-for record from durable storage before participating in any election.
+3. When a persisted term file does not exist at startup (first boot), the system shall initialize the term to zero with no prior vote recorded.
+
+---
+
+### Requirement 10: Election and Replication Observability
+
+**Objective:** As a cluster operator, I want election results, vote tallies, config bootstrap operations, WAL commit quorum status, and config push quorum status to appear in structured logs at INFO or WARN severity, so that cluster health and HA behavior can be monitored without external tooling.
+
+#### Acceptance Criteria
+1. When an election completes, the system shall log at INFO level the outcome (primary won or standby), the term, the number of votes granted, and the quorum size.
+2. When a config bootstrap probe determines the config situation (all peers tag zero, config exists on peers, or no peer reachable), the system shall log the finding at INFO or WARN level.
+3. When a WAL commit achieves quorum, the system shall log at INFO level; when a WAL commit times out, the system shall log at WARN level.
+4. When a configuration upgrade achieves quorum, the system shall log at INFO level; when a configuration upgrade times out, the system shall log at WARN level.
+5. When a controller transitions between the init, electing, primary, and standby states, the system shall log the transition at INFO level.

.kiro/specs/ha-config-bootstrap/research.md

@@ -0,0 +1,277 @@
+# Gap Analysis: ha-config-bootstrap
+
+*Generated: 2026-05-27. Branch: persistent_ha.*
+
+---
+
+## Analysis Summary
+
+- **Scope**: 10 requirements across 8 distinct changes (A–H). The codebase has strong foundational infrastructure (iomgr, repl slot management, WAL, SSL) but the HA election core, quorum commit, and control channel are mostly absent or stubbed.
+- **Largest gaps**: Raft-style quorum election (Req 2) and persistent control channel with full state sync (Req 6) are full rewrites/additions with no current equivalent logic; majority-commit WAL (Req 5) requires implementing the two stub functions at the heart of replication.
+- **Strong reusable assets**: `repl_add_standby/remove/start_catchup/complete_catchup/needs_snapshot` are real implementations; the SSL connect pattern (`connect_ssl_raw`), xorshift64 PRNG, `ha_do_promote/demote`, and iomgr epoll machinery are all directly reusable.
+- **Risk concentration**: Req 2 and Req 6 carry High risk; Req 5 carries High risk due to `committed_lsn` cross-cutting `persist_t`, `repl.c`, and `ha_state_t`. The remaining requirements are Medium or Low.
+- **Recommended approach**: Hybrid (Option A) — implement changes A→D→B→C→E→F→G→H in dependency order, writing C tests first per TDD, one atomic commit per change.
+
+---
+
+## 1. Current State Investigation
+
+### 1.1 HA Core (ha.h / ha.c)
+
+**`ctrl_role_t` enum** — only two values: `CTRL_PRIMARY = 0`, `CTRL_STANDBY = 1`.  
+Missing: `CTRL_INIT`, `CTRL_ELECTING` (required by Req 1).
+
+**`ha_state_t`** — current fields of interest:
+
+| Field | Status | Notes |
+|---|---|---|
+| `term` (_Atomic uint64_t) | Exists | Used for fencing |
+| `role` (_Atomic int32_t) | Exists | Cast to ctrl_role_t |
+| `my_priority` | Exists | LSF priority index — kept for compat but unused in Raft |
+| `applied_lsn` (_Atomic uint64_t) | Exists | Reusable for vote freshness check |
+| `primary_host[XCE_MAXNAME_LEN]` | Exists | Reusable |
+| `primary_port` | Exists | Reusable |
+| `last_heartbeat` (struct timespec) | Exists | Replaces with `EPOLLRDHUP` path in Req 6 |
+| `in_catch_up` (_Atomic int) | Exists | Reusable for Clarification 6 vote exclusion |
+| `jitter_seed` (uint64_t) | Exists | xorshift64, directly reusable |
+| `ssl_ctx` (void*) | Exists | Reusable |
+| `standby_endpoints` (zarray_t*) | Exists | Superseded by per-standby fd table in Req 6 |
+| `voted_for_term` (uint64_t) | **MISSING** | Req 9 |
+| `voted_for_name` (char[]) | **MISSING** | Req 9 |
+| `primary_ctrl_fd` (int) | **MISSING** | Req 6 |
+| `defer_config_fetch` (bool) | **MISSING** | Req 3 |
+| `elect_ctx` (_Atomic ptr) | **MISSING** | Req 2 |
+| `last_known_committed_lsn` (_Atomic uint64_t) | **MISSING** | Req 5 |
+| `heartbeat_seen` (sem_t) | **MISSING** | Req 1 |
+
+**`ha_config_t`** — missing: `commit_timeout_ms`. All election fields present (election_timeout_ms, jitter_max_ms, etc.) but values tuned for LSF model.
+
+**`am_i_primary()`** — LSF priority probe via per-peer pthreads. No quorum, no config_tag/LSN in vote, no term/voted_for persistence. Must be replaced entirely with `am_i_primary_quorum()`.
+
+**`ha_check_priority_stepdown()`** — permanently unreachable once term > 0; exists only for the term=0 simultaneous-boot race. Must be retired (Req 2).
+
+**`ha_check_heartbeat_nocluster()`** — timestamp polling with configurable timeout. Replaced by `EPOLLRDHUP` fast path + heartbeat timeout vtask (Req 6). Function itself can be deleted.
+
+**`ha_watchdog_thread()`** — sets `failover_requested` but cannot call `ha_promote_to_primary()` (no `cluster_t`). Partially functional stub. Replaced by vtask-based heartbeat monitor (Req 6).
+
+**`start_heartbeat_listener()` / `stop_heartbeat_listener()`** — no-op stubs with comment "full implementation in cluster.c Task 7". **Missing** (Req 1).
+
+**`ha_fini()`** — missing `SSL_CTX_free(ha->ssl_ctx)`. ASAN-visible leak (~5 KB).
+
+### 1.2 Startup Sequence (myapp.c / main.c)
+
+**myapp.c current order**:
+```
+cluster_init_phase1()
+initialize_openssl() + create_ssl_context()
+am_i_primary()                     ← election here
+load_primary_config() / fetch_config_from_primary()
+cluster_init_phase2()              ← iomgr, vtaskmgr, bind+listen inside
+cluster_start()                    ← more network init
+cluster_run()
+```
+
+**Bind+listen** happens inside `cluster_init_phase2()` or `cluster_start()` — **after** election. Gap: concurrent-startup nodes probe an unbound port (Req 1).
+
+**No `bootstrap_config_if_needed()`** — standby path goes straight to `fetch_config_from_primary()` post-election without checking whether local config_tag is 0 or whether peers have config. Gap (Req 3).
+
+**main.c**:
+- `--vcpus N` arg: **MISSING**. vcpus comes from cluster config read inside phase2. This creates a dependency that prevents vtaskmgr from running before election (Req 1, 4).
+- Peer count validation: **MISSING**. Even-N peer lists accepted silently (Req 4).
+- Cert pre-flight `access()` check: **MISSING**. Cert paths computed but SSL context created later in myapp.c, not with early-exit error in main.c (Req 4).
+
+### 1.3 Replication (repl.c / repl.h)
+
+| Function | Status | Notes |
+|---|---|---|
+| `repl_init / fini` | Real | Lifecycle ok |
+| `repl_add_standby` | Real | Slot management implemented |
+| `repl_remove_standby` | Real | Implemented |
+| `repl_start_catchup` | Real | Sets in_catch_up=1 |
+| `repl_complete_catchup` | Real | Clears in_catch_up, broadcasts cond |
+| `repl_needs_snapshot` | Real | Compares standby_lsn vs oldest_wal_lsn |
+| `repl_on_ack` | Real | Updates acked_lsn, signals cond |
+| `repl_trim_lsn` | Real | WAL trim guard implemented |
+| `repl_broadcast` | **STUB** | Returns 0 without sending anything (Req 5, 8) |
+| `repl_wait_ack` | Partial | Waits for ANY non-catch-up standby, not majority (Req 5) |
+| `repl_wait_majority_ack` | **MISSING** | Req 5, 8 |
+
+**`standby_conn_t`** — missing `pending_ack_count` for backpressure (Req 5).
+
+### 1.4 Persistence (persist.h / persist.c)
+
+- `current_lsn` and `checkpoint_lsn` exist. **`committed_lsn`** — **MISSING** (Req 5).
+- `persist_wal_sync_append()` uses `repl_wait_ack()` (any-standby) — must upgrade to `repl_wait_majority_ack()`.
+- `persist_truncate_wal_above()` — **MISSING** (Req 5, 6).
+- `persist_config_t.sync_timeout_ms` exists — analogous to `commit_timeout_ms` needed for majority-commit.
+
+### 1.5 Config Replication (config_repl.c)
+
+- `do_one_config_fetch()` — **wired** (recent commit `2aae45f` on branch: "wire do_one_config_fetch() — full SSL roundtrip"). Not a stub.
+- `bootstrap_config_if_needed()` — **MISSING** (Req 3).
+- `xce_probe_config_tag()` — **MISSING**. New lightweight RPC needed (Req 3).
+- `push_config_to_standbys()` — declared in config_repl.h; likely fire-and-forget (no majority wait). **Gap**: must upgrade to majority-commit (Req 8).
+- `OP_probeConfigTag` Cap'n Proto schema — **MISSING** (Req 3).
+- `OP_upgradeConfigAck` Cap'n Proto schema — **MISSING** (Req 8).
+- `committed_config_tag` — **MISSING** (Req 8).
+
+### 1.6 Control Channel (ha.c / cluster.c)
+
+- No persistent standby-to-primary TCP connection. Each config fetch and WAL entry opens a new SSL connection via `connect_ssl_raw()`.
+- `ha_find_and_connect_to_primary()` — **MISSING** (Req 6).
+- Extended `OP_ctrlHello` with applied_lsn, config_tag — **MISSING** (Req 6).
+- `OP_snapshotTransfer`, `OP_walCatchupEntry` — **MISSING** (Req 6).
+- `handle_ctrlhello()` extended handler — **MISSING** (Req 6).
+- Three-tier primary discovery — **MISSING** (Req 6).
+- EPOLLRDHUP failover detection — **MISSING** (Req 6).
+- Heartbeat timeout vtask replacing watchdog thread — **MISSING** (Req 6).
+
+### 1.7 Election State Durability
+
+- `ha-term.bin` file — **does not exist**.
+- `ha_persist_term()` — **MISSING** (Req 9).
+- `ha_init()` zeros term and has no `ha-term.bin` read path.
+
+### 1.8 libxce Client Retry (src/lib/)
+
+- `ha_should_redirect()` / `ha_no_primary_known()` exist in ha.c and are used by join.c.
+- No transparent retry loop for `XCE_ERR_NOT_READY`, `XCE_ERR_STANDBY`, `ECONNREFUSED`.
+- `xce_retry_config_t` — **MISSING** (Req 7).
+- `redirect_host` / `redirect_port` fields in `XCE_ERR_STANDBY` error reply — **MISSING** (Req 7).
+
+---
+
+## 2. Requirement-to-Asset Map
+
+| Req | Key Gap | Key Reusable Asset | Gap Tag |
+|---|---|---|---|
+| 1 — Listener before election | `CTRL_INIT`/`CTRL_ELECTING` enum values; pre-dispatch gate in vtaskmgr; extract bind+listen from phase2; `ha_wait_for_heartbeat_or_timeout()`; `heartbeat_seen` semaphore | iomgr epoll, vtaskmgr dispatch switch, xorshift64 jitter | **Missing** |
+| 2 — Quorum election | Entire `am_i_primary_quorum()` and vote protocol; `ha_vote_entry_t`, `ha_elect_ctx_t`; `handle_request_vote()` / `_reply()`; `ha_elect_evaluate_and_signal()`; OP_requestVote capnp schema | `ha_do_promote/demote`, jitter PRNG, `applied_lsn`, iomgr async connect | **Missing** |
+| 3 — Config bootstrap | `bootstrap_config_if_needed()`; `xce_probe_config_tag()`; OP_probeConfigTag capnp; `defer_config_fetch` flag in ha_state_t | `do_one_config_fetch()` (wired), `connect_ssl_raw()` | **Missing** |
+| 4 — Startup validation | `--vcpus N` CLI arg; even-N peer count check; cert pre-flight `access()`; `SSL_CTX_free` ASAN fix | argp option parsing pattern in main.c | **Missing** |
+| 5 — Majority-commit WAL | `repl_broadcast()` implementation; `repl_wait_majority_ack()`; `committed_lsn` in persist_t; `persist_truncate_wal_above()`; `committed_lsn` in heartbeat; `last_known_committed_lsn` in ha_state_t | `repl_wait_ack()` logic, `standby_conn_t`, `repl_start/complete_catchup`, WAL sync path | **Missing** / **Stub** |
+| 6 — Persistent channel | Persistent TCP fd per standby; `ha_find_and_connect_to_primary()`; EPOLLRDHUP failover; extended OP_ctrlHello; OP_snapshotTransfer; OP_walCatchupEntry; WAL streaming loop; heartbeat vtask; three-tier discovery | iomgr EPOLLIN/EPOLLRDHUP, `repl_needs_snapshot()`, `repl_start/complete_catchup()` | **Missing** |
+| 7 — Client retry | Retry loop in libxce; `xce_retry_config_t`; redirect fields in error reply | `ha_should_redirect()`, `ha_no_primary_known()` | **Missing** |
+| 8 — Majority-commit config | `repl_wait_majority_ack()` (shared with Req 5); OP_upgradeConfigAck; `committed_config_tag`; heartbeat piggybacking | `push_config_to_standbys()` (fire-and-forget → upgrade) | **Missing** |
+| 9 — State durability | `ha_persist_term()`; `ha-term.bin` read in `ha_init()`; `voted_for_term`/`voted_for_name` in ha_state_t | Atomic write+rename pattern (existing in config_repl.c) | **Missing** |
+| 10 — Observability | Structured log statements throughout all new code paths | Existing INFO/WARN/ERROR macros (xcelog.h) | **Missing** (alongside) |
+
+---
+
+## 3. Implementation Approach Options
+
+### Option A: Change-at-a-time (recommended)
+
+Implement changes in dependency order: D → A → B → C → E → H → F → G → (observability woven in).
+
+**Rationale for ordering**:
+1. D first: `--vcpus N` removes vtaskmgr's config dependency, unblocking A.
+2. A before B: listener must be open before election dispatch works.
+3. B before C: config bootstrap needs the proven election winner from quorum election.
+4. E and H together: both use `repl_broadcast()` + `repl_wait_majority_ack()`; implement infrastructure once.
+5. F after E: persistent channel carries WAL entries; broadcast must send over that channel.
+6. G last: retry makes sense only after the controller states it retries against are implemented.
+
+**Files touched per change**:
+- D: `main.c`, `cluster.c` (remove vcpus from config), `ha.c` (`SSL_CTX_free`)
+- A: `ha.h` (enum + semaphore + fields), `myapp.c` (startup reorder), `cluster.c` (extract listener), `ha.c` (wait function, dispatch gate)
+- B: `ha.h` (new types), `ha.c` (replace `am_i_primary`), capnp schema, `cluster.c` (dispatch handler)
+- C: `config_repl.c` (`bootstrap_config_if_needed`, `xce_probe_config_tag`), capnp schema, `myapp.c` (integration)
+- E: `repl.c` (`repl_broadcast`, `repl_wait_majority_ack`), `persist.h`/`persist.c` (`committed_lsn`, `persist_truncate_wal_above`), `ha.h`/`ha.c` (`last_known_committed_lsn`)
+- H: `config_repl.c` (upgrade majority-commit), capnp schema, `ha.h` (`committed_config_tag`)
+- F: `ha.c`/`cluster.c` (channel connect, extended hello, streaming, EPOLLRDHUP)
+- G: `src/lib/` (retry loop, xce_retry_config_t, redirect fields)
+
+**Trade-offs**:
+- ✅ Each atomic commit is testable before the next starts
+- ✅ Follows existing TDD convention (failing test first)
+- ❌ Long sequential dependency chain; F and G can only start late
+
+### Option B: Infrastructure-first parallel tracks
+
+Track 1 (election+bootstrap): D → A → B → C  
+Track 2 (replication): E → H  
+Join (channel): F using Track 1 + Track 2 artifacts  
+Final: G
+
+**Trade-offs**:
+- ✅ Track 1 and Track 2 are mostly independent until F
+- ❌ Track 2 cannot be fully validated without Track 1 (no quorum without election)
+- ❌ More coordination overhead between developers on shared ha.h
+
+### Option C: Stub-forward then wire
+
+Add all new stubs with `-ENOSYS` first, then wire each one. Matches the existing codebase pattern (many stubs already present).
+
+**Trade-offs**:
+- ✅ Compiles throughout; no long broken-build windows
+- ❌ Stub-first delays real testability; TDD mandate disfavors this
+
+**Recommendation**: Option A. The dependency chain is well-defined and single-developer serial implementation is the realistic mode for this C codebase.
+
+---
+
+## 4. Effort and Risk Summary
+
+| Req | Effort | Risk | Justification |
+|---|---|---|---|
+| 1 — Listener before election | M (3–5d) | Medium | Refactor startup order; must remove vtaskmgr/config dependency first |
+| 2 — Quorum election | XL (2+ wks) | High | Full algorithm replacement; new wire protocol; FSM extension; concurrent dispatch |
+| 3 — Config bootstrap | M (3–5d) | Medium | New function + probe RPC; depends on do_one_config_fetch (wired) |
+| 4 — Startup validation | S (1d) | Low | Additive CLI + validation; ASAN one-liner |
+| 5 — Majority-commit WAL | L (1–2 wks) | High | `committed_lsn` cross-cuts persist+repl+ha; `repl_broadcast` must become real |
+| 6 — Persistent channel | XL (2+ wks) | High | New connection lifecycle; full state sync protocol; 3-tier discovery |
+| 7 — Client retry | M (3d) | Low | Well-understood retry pattern; additive libxce change |
+| 8 — Majority-commit config | M (3d) | Medium | Shares Req 5 infrastructure; mainly wiring push_config_to_standbys |
+| 9 — State durability | S (1d) | Low | File write+read; extends ha_init |
+| 10 — Observability | S (alongside) | Low | Log statements woven into each change |
+
+**Total estimated effort**: 6–9 weeks for one engineer implementing with full TDD.
+
+**High-risk dependencies**:
+- Req 2 must complete before Req 3 can be integrated end-to-end.
+- Req 5 (`repl_broadcast` implementation) must complete before Req 8 and before Req 6 (WAL streaming over the channel).
+- Req 4 (vcpus CLI) must complete before Req 1 (vtaskmgr before election).
+
+---
+
+## 5. Key Reusable Assets
+
+| Asset | Location | Reuse in |
+|---|---|---|
+| `repl_add/remove_standby` | repl.c | Req 6 (channel registration) |
+| `repl_start/complete_catchup` | repl.c | Req 6 (state sync protocol) |
+| `repl_needs_snapshot` | repl.c | Req 6 (snapshot decision) |
+| `repl_on_ack` | repl.c | Req 5 (majority-commit ack handler) |
+| `repl_trim_lsn` | repl.c | Req 5 (WAL trim with committed_lsn) |
+| `connect_ssl_raw()` | netio.c/config_repl.c | Req 3 (probe RPC) |
+| `do_one_config_fetch()` | config_repl.c | Req 3 (deferred fetch) |
+| `xorshift64_next + ha_seed_jitter` | ha.h / ha.c | Req 1, 2 (jitter) |
+| `ha_do_promote / ha_do_demote` | ha.c | Req 2, 6 (role transitions) |
+| `ha_should_redirect / ha_no_primary_known` | ha.c | Req 7 (retry redirect) |
+| iomgr epoll + EPOLLIN/EPOLLRDHUP | iomgr (cluster.c) | Req 1, 6 |
+| Atomic write+rename pattern | config_repl.c | Req 9 (ha-term.bin) |
+| INFO/WARN/ERROR macros | xcelog.h | Req 10 |
+
+---
+
+## 6. Constraints for Design Phase
+
+- **C gnu17, no C++, no new external dependencies**.
+- ASAN build (`nix build .#xcecore_debug`) must pass after every change.
+- **Static peer membership**: `--peers` list is fixed at startup; no dynamic join/leave of controller peers. Quorum denominator is fixed for cluster lifetime.
+- **TDD mandatory**: failing test in `src/controller/testsuite/` before each implementation change.
+- `make test` must pass without a live server at every commit.
+- `commit_timeout_ms` must be configurable (default 5000ms).
+- **Single-node mode** (no `--peers`): quorum=1 everywhere; all new majority-commit paths must short-circuit to self-only.
+- **WAL write stall window** (operational constraint, not a bug): during standby catch-up after election, `repl_wait_majority_ack()` will time out on every write until at least one standby completes catch-up. This is correct CP behavior; operators must be warned in docs.
+
+---
+
+## 7. Research Needed (Design Phase)
+
+- **vtaskmgr API for pre-election dispatch**: confirm that `vtaskmgr_init(vcpus)` can be called before `cluster_config` is loaded (after removing vcpus from config struct). Verify no hidden config dependency.
+- **iomgr async connect pattern for vote requests**: confirm `iomgr_async_vote_request()` can be implemented using existing async connect infrastructure without new socket primitives.
+- **capnp schema extension**: confirm the capnp code-generation toolchain can add `OP_requestVote`, `OP_probeConfigTag`, `OP_walAck`, `OP_upgradeConfigAck`, `OP_snapshotTransfer`, `OP_walCatchupEntry` without breaking existing decode paths.
+- **`persist_truncate_wal_above()` safety**: verify WAL segment file structure supports efficient truncation at arbitrary LSN boundary without full segment rewrite.
+- **libxce client API surface**: identify the exact function in src/lib/ that wraps xce_encode()+send+recv+xce_decode() to locate the correct insertion point for retry logic.

.kiro/steering/error-handling.md

@@ -0,0 +1,92 @@
+# Error Handling Standards
+
+## Philosophy
+
+Fail fast with a consistent, machine-readable response. Handle errors at the controller boundary; let business logic return typed errors. Never silently swallow errors.
+
+## Go API Response Pattern
+
+All HTTP responses go through two helpers in `controllers/response.go`:
+
+```go
+// Success (always HTTP 200)
+ReturnSuccess(ctx, common.SUCCESS_CODE, common.SUCCESS_MSG, data)
+
+// Error (HTTP status == code parameter)
+ReturnError(ctx, common.ERROR_AUTH_CODE, common.ERROR_ACCESS_FORBIDDEN_MSG, nil)
+ctx.Abort()   // always Abort() after ReturnError in middleware
+return        // always return immediately after ReturnError in handlers
+```
+
+**The early-return pattern is non-negotiable.** Every error path ends with `ReturnError(...); return`.
+
+## HTTP Status Code Mapping
+
+| Code | Constant | When to use |
+|------|----------|-------------|
+| 200 | `SUCCESS_CODE` | Successful operation |
+| 201 | `CREATED` | Resource created |
+| 400 | `INVALID_PARAMS_CODE` | Malformed or invalid input |
+| 401 | `ERROR_AUTH_CHECK_TOKEN_CODE` | Missing or invalid JWT |
+| 403 | `ERROR_AUTH_CODE` | Valid JWT but insufficient permission |
+| 404 | `ERROR_NOT_FOUND_CODE` | Resource not found |
+| 429 | `ERROR_TOO_MANY_REQUESTS_CODE` | Rate limit exceeded |
+| 500 | `ERROR_CODE` | Unexpected server error |
+
+All code constants are defined in `common/common.go`. Add new codes there — do not inline numeric literals in handlers.
+
+## Structured Logging with Zap
+
+Log at the call site before returning an error. Use typed field helpers — never `fmt.Sprintf` in zap calls.
+
+```go
+// Preferred pattern
+logger.Error("rate limit exceeded",
+    zap.String("path", path),
+    zap.String("client_ip", ctx.ClientIP()),
+    zap.Int("errorcode", code),
+)
+
+// For error values
+logger.Error("validate token error", zap.Error(err))
+
+// For complex types
+logger.Debug("permissions resolved", zap.Any("permsWithType", permsWithType))
+```
+
+**Log levels**:
+- `Error` — operation failed, client or server fault
+- `Warn` — recoverable edge case (e.g., retried operation succeeded)
+- `Info` — key lifecycle events (server start, config load)
+- `Debug` — diagnostic detail, disabled in production
+
+**Never log**: JWT tokens, passwords, secret keys, or full request bodies containing sensitive data.
+
+## C Error Propagation
+
+Return negative errno-style codes for C daemon errors. Use the `errnox.h` definitions from `src/include/xce/`. Check and propagate immediately — do not continue on error:
+
+```c
+int rc = xce_send_request(&req);
+if (rc < 0) {
+    log_error("send_request failed: %d", rc);
+    return rc;
+}
+```
+
+## Input Validation Pattern
+
+Validate at the controller boundary using a `preCheck{Resource}Request()` helper before touching services:
+
+```go
+func (c JobController) SubmitJob(ctx *gin.Context) {
+    req, err := preCheckSubmitJobRequest(ctx)
+    if err != nil {
+        ReturnError(ctx, common.INVALID_PARAMS_CODE, err.Error(), nil)
+        return
+    }
+    // ... service call
+}
+```
+
+Keep `preCheck*` functions in the same file as the handler. They bind, validate, and return the typed request struct or an error.

.kiro/steering/product.md

@@ -0,0 +1,43 @@
+# Product Overview
+
+XCE is an HPC (High-Performance Computing) cluster management system providing job scheduling, queue management, and cluster resource control. It targets organizations running batch workloads on on-premises or air-gapped compute clusters who need deterministic scheduling, multi-tenant access control, and zero external database dependencies.
+
+## Core Capabilities
+
+1. **Job lifecycle management** — submit, start, stop, restart, kill, and monitor jobs including job arrays. Resource usage data is collected and exposed via API.
+2. **Queue and scheduling** — plugin-based scheduling engine for job-to-node placement with queue control (enable/disable) and expression-based resource filtering.
+3. **Cluster and host management** — node status monitoring (online/offline/unavailable), cluster topology tracking, and host control operations.
+4. **Multi-tenant RBAC + pluggable auth** — domain-based `action:scope` permission model (e.g., `job:view:self`). Auth backends: configfile, LDAP, OAuth2, PAM. Built-in IP whitelist and rate limiting.
+5. **Controller persistence and HA** — WAL + checkpoint-based single-controller persistence with LSF-style priority election for standby failover and live replication over TCP (Cap'n Proto serialization).
+
+## Target Use Cases
+
+- **Local HPC batch computing** — long-running scientific/engineering workloads requiring deterministic scheduling, queue priority, and resource isolation.
+- **Air-gapped / secure environments** — standalone deployments with file-based auth + SSL, no external cloud dependencies.
+- **Multi-team shared clusters** — multiple teams sharing a cluster through RBAC with JWT token authentication.
+- **No-shared-storage HPC** — controller uses local filesystem (ext4/xfs) for storage; job files transferred via API upload/download rather than network shares.
+
+## Value Proposition
+
+XCE provides a **fully integrated single-vendor stack** (no database setup, unified build system) with an open, buildable codebase (Nix flakes, AppImage packaging). Compared to Slurm/PBS, it eliminates the external database requirement. Compared to commercial LSF, it is open source and reproducibly buildable.
+
+**Key architectural trade-off**: single-writer model throughout — no multi-master consensus. HA failover uses LSF-style priority election (simple, predictable) rather than Raft (10× complexity). All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
+
+## Active Development Context
+
+The project is in a **persistence and HA transformation** (branch `persistent_ha`). Phases:
+
+| Phase | Goal | Status |
+|-------|------|--------|
+| P0 | Restart survival (WAL + checkpoint) | Implementing |
+| P1 | Standby election and takeover | Designed |
+| P2 | Live replication primary → standby | Designed |
+| P2.5 | Worker/scheduler auto-reconnect on failover | Designed |
+| P3–P4 | Hardening, metrics, docs | Planned |
+
+Key design decisions already locked:
+- No `fork()` for snapshots (mutex inheritance deadlock in multithreaded controller)
+- No Raft (single writer makes consensus overhead unjustified)
+- No BoltDB (using dual zarray COW for application-level checkpointing)
+- No application-layer encryption (trust OS boundary: chmod 0700, LUKS/dm-crypt)
+- No WAL logging for job deletion (deferred to checkpoint time to simplify replay)

.kiro/steering/security.md

@@ -0,0 +1,61 @@
+# Security Standards
+
+## Philosophy
+
+Defense in depth, least privilege, fail closed. Authentication verifies *who*; authorization verifies *what they can do*. Centralize both — never duplicate permission checks across handlers.
+
+## RBAC Enforcement
+
+All protected routes use `RequirePermission` middleware attached at registration time:
+
+```go
+// In router/routers.go
+hostsGroup.GET("", RequirePermission("host:view"), handler)
+hostsGroup.POST("/control", RequirePermission("host:control"), handler)
+```
+
+`RequirePermission` in `controllers/rbac_check.go` resolves the user from the JWT, looks up their effective permissions (including inherited role permissions), and aborts with 403 if the required permission is absent.
+
+**cluster_admin** bypasses all permission checks via early return — do not add new bypass paths outside this single mechanism.
+
+## Permission String Format
+
+`"action:scope"` — e.g., `"job:submit"`, `"host:view"`, `"queue:manage"`.
+
+Scopes: `all` (unrestricted), `self` (own resources), `cluster`, `hostgroup`, `self_ug`, `self_hg`.
+
+Permissions are defined in `config/rbac.yaml`. Roles support inheritance via `includes:` — resolution is recursive. Add new permissions to the YAML first, then wire them onto routes.
+
+## JWT Authentication
+
+Token flow:
+1. `POST /auth/logon` → validates credentials via configured auth backend → issues JWT with `{username, IP, expiry}` claims
+2. Every subsequent request: `Authorization: Bearer <token>` header
+3. `middleware/validatehandler.go` strips the prefix, parses with golang-jwt, **validates the IP claim against the request IP**
+
+IP-bound tokens prevent token theft across network boundaries. Do not remove the IP validation.
+
+Token secret is loaded from `config/auth.yaml` at startup. It must never appear in source code or logs.
+
+## Auth Backends
+
+Backend selection is controlled by `config/auth.yaml`. Supported: `configfile`, `ldap`, `oauth2`, `pam`. All backends implement the same interface — adding a new backend means implementing that interface, not modifying middleware.
+
+## Input Validation
+
+Validate at the API boundary inside `preCheck{Resource}Request()` helpers — before any service or store access. Return 400 with a clear message on validation failure. Do not pass unvalidated input to downstream layers.
+
+## Secrets
+
+- JWT secret key: `config/auth.yaml` (file-system protected, chmod 0600)
+- LDAP bind credentials: `config/auth.yaml`
+- **Never log** tokens, passwords, secret keys, or full request bodies containing credentials
+- Validate all required config at startup; fail fast with a clear error if missing
+
+## Rate Limiting and IP Whitelist
+
+Rate limiting (`middleware/rate_limit.go`) and IP whitelist (`middleware/ip_whitelist.go`) are applied before JWT validation in the middleware chain. They use `ctx.ClientIP()` and abort with 429 / 403 respectively. Both are configured in `config/` YAML — do not hardcode thresholds.
+
+## Transport
+
+All daemon-to-daemon communication is over TCP. Production deployments add TLS at the network layer (not handled in application code). The WAL and replication wire format uses Cap'n Proto — schema files in `src/schema/`. Do not add unvalidated deserialization paths for external input.

.kiro/steering/structure.md

@@ -0,0 +1,99 @@
+# Project Structure
+
+## Organization Philosophy
+
+**Layered monorepo, component-per-directory.** Each major subsystem lives under `src/<component>/` with its own Meson build file and CLAUDE.md. The Go frontend is isolated in `src/frontend/` as two independent Go modules. All C daemons share `src/lib/` (libxce) and public headers from `src/include/xce/`.
+
+```
+src/
+├── lib/           C — shared library (libxce): data structures, IPC, utilities
+├── controller/    C — cluster controller daemon
+├── worker/        C — per-node job execution daemon (xcecd)
+├── scheduler/     C — job scheduling engine (plugin architecture)
+├── autoscaler/    C — autoscaling engine (provider plugins)
+├── frontend/      Go — apiserver (Gin REST) + xcecli (Cobra CLI)
+├── include/xce/   C — public shared headers consumed across all daemons
+├── schema/        Cap'n Proto schema files (c.capnp, controller.capnp)
+└── docs/          Project-wide documentation (markdown)
+```
+
+## C Daemon Layout Pattern
+
+Each C daemon (`controller/`, `worker/`, `scheduler/`, `autoscaler/`) follows:
+
+```
+src/<component>/
+    meson.build          — build definition
+    CLAUDE.md            — component-specific AI guide
+    include/             — component-internal headers
+    src/                 — .c implementation files (flat, one domain per file)
+    testsuite/           — test programs (test_*.c)
+```
+
+**Exception**: `controller/` keeps its `.c` files flat in the component root (not in a nested `src/`). All other C daemons use `src/<component>/src/`.
+
+Plugin-based components add their own subdirectory:
+- `scheduler/` — plugins in `src/plugins/`
+- `autoscaler/` — providers in `src/autoscaler/providers/`
+
+## Go Frontend Layout Pattern
+
+**apiserver** (`src/frontend/apiserver/`):
+
+| Package | Purpose |
+|---------|---------|
+| `cmd/` | Entry point, init sequence |
+| `router/` | Route registration + middleware chain |
+| `controllers/` | HTTP handlers — one file per resource |
+| `services/` | Business logic |
+| `store/` | Thread-safe in-memory KV stores (generics) |
+| `models/` | Request/response DTOs, domain structs |
+| `middleware/` | CORS, rate limiting, JWT, IP whitelist |
+| `backend/` | CGO bridge to C library |
+| `backend/auth/` | Pluggable auth providers |
+| `utils/` | Zap logger wrapper, config parsing, crypto |
+| `common/` | Global error codes and response helpers |
+| `config/` | YAML configs + JSON schemas |
+
+## Naming Conventions
+
+### C
+
+- **Files**: `snake_case.c` / `snake_case.h` — one domain per file
+- **Public headers**: `src/include/xce/<name>.h` — the libxce API boundary
+- **Component headers**: `src/<component>/include/<name>.h`
+- **Test files**: `test_<unit>.c` (e.g., `test_ha_cluster.c`, `test_persist_wal.c`)
+- **Functions**: `snake_case` with component prefix — `ha_*`, `cluster_*`, `xcecd_*`, `xce_*`
+- **Types**: `snake_case_t` suffix (e.g., `cluster_t`, `ha_state_t`)
+- **Struct tags**: `struct <name>_s` (e.g., `struct cluster_s`)
+- **Enums/Macros**: `UPPER_SNAKE_CASE` with component prefix (`CTRL_PRIMARY`, `HA_DEFAULT_*`)
+
+### Go
+
+- **Files**: one file per resource — `jobs.go`, `queues.go`, `hosts.go`
+- **Tests**: `*_test.go` co-located; integration tests as `integration_test.go`, `<resource>_control_test.go`
+- **Controller structs**: `{Resource}Controller`
+- **Handler methods**: `{Action}{Resource}` (e.g., `SubmitJob`, `ListJobs`)
+- **Permission strings**: `"action:scope"` (e.g., `"job:view:self"`)
+- **JSON tags**: `snake_case`
+
+## Test Organization
+
+**C tests** (`testsuite/` inside each component):
+- Standalone `.c` programs, one per unit: `test_<unit>.c`
+- `test.c` provides the test runner harness
+- Tiered naming: `test_p1_integration.c`, `test_e2e_wal.c`, `test_p3_hardening.c`
+- Test-only symbols guarded by `#ifdef XCE_TESTING`
+
+**Go tests** (co-located `_test.go`):
+- Table-driven unit tests, `t.TempDir()` for fixtures
+- Global state reset helpers (e.g., `resetRBACState()`) for test isolation
+- `make test` = unit only; `make test-all` = unit + integration
+
+## Code Organization Principles
+
+- **One domain per file** in C — `ha.c` owns HA logic, `job.c` owns job logic. No megafiles.
+- **Single Go module per binary** — apiserver and xcecli share no Go code at the module level.
+- **Controllers delegate to services** — HTTP handlers validate input and delegate; business logic lives in `services/`.
+- **Dependencies flow inward**: `controllers → services → store/backend`, never the reverse.
+- **Cap'n Proto for wire formats** — any new inter-daemon or persistence protocol uses `.capnp` schemas in `src/schema/`.

.kiro/steering/tech.md

@@ -0,0 +1,100 @@
+# Technology Stack
+
+## Architecture
+
+```
+xcecli (Go/Cobra) ──REST──► apiserver (Go/Gin) ──CGO──► C daemon layer
+                                                          controller / scheduler / worker / autoscaler
+                                                          ↑↑↑ all linked against libxce (src/lib/)
+```
+
+Request path inside apiserver: `Gin Router → Middleware (CORS/RateLimit/IPWhitelist/JWT) → RBAC Check → Controller → Services → In-Memory Store`
+
+All controller state is in-memory, seeded at startup from YAML config. No persistent database in the API tier. C daemons communicate via libtirpc RPC; WAL + checkpoint provide persistence at the controller level.
+
+## Core Technologies
+
+- **Go 1.23.4** — apiserver (REST API) and xcecli (CLI). Two independent Go modules, no `go.work`.
+- **C (gnu17)** — controller, worker, scheduler, autoscaler, libxce. No C++.
+- **Gin v1.10.0** — HTTP router, middleware, binding
+- **Cobra v1.9.1** — CLI framework for xcecli
+- **Zap v1.27.0** — structured logging throughout (both Go modules)
+- **Viper v1.19.0 / v1.20.1** — YAML config management
+- **Cap'n Proto** — wire serialization for WAL and daemon-to-daemon replication
+- **libtirpc** — inter-daemon RPC transport
+
+## Build Systems
+
+| System | Scope |
+|--------|-------|
+| **Meson ≥ 1.5** | C daemons + Go modules via `tools/go_build` wrapper |
+| **Make** | Go development loop (build/test) from `src/frontend/` |
+| **Nix flakes** | Hermetic dev environment, AppImage packaging |
+
+```bash
+# Full build (C + Go, debug + ASAN)
+/xce-build
+
+# Go development loop (from src/frontend/)
+make build-server          # → /tmp/apiserver
+make build-cli             # → /tmp/xce_static (CGO_ENABLED=0)
+make test                  # unit tests
+make test-integration      # regression suite (needs XCE_BIN + SRV_URL)
+
+# Nix package builds (from src/)
+nix build .#xcecore        # C daemons + apiserver
+nix build .#xcecore_debug  # debug + ASAN
+nix build .#xcecli         # CLI only
+```
+
+## Development Standards
+
+### Controller Pattern (Go)
+
+```go
+// Struct: {Resource}Controller
+type JobController struct { ... }
+
+// Handler: {Action}{Resource}
+func (c JobController) SubmitJob(ctx *gin.Context) {
+    // Validate input
+    req, err := preCheckSubmitJobRequest(ctx)
+    if err != nil { ReturnError(ctx, code, err, nil); return }
+
+    // Business logic
+    result, err := c.svc.Submit(req)
+    if err != nil { ReturnError(ctx, code, err, nil); return }
+
+    ReturnSuccess(ctx, code, "submitted", result)
+}
+```
+
+Early return on every error. Log with zap fields before returning.
+
+### Testing Requirements
+
+- **TDD mandatory**: Write failing test first, then implementation.
+- **Go unit tests**: Table-driven, no external dependencies, `t.TempDir()` for fixtures.
+- **C tests**: Add test case in `testsuite/` before modifying implementation.
+- `make test` must pass without a live server.
+- `make test-integration` requires `XCE_BIN` + `SRV_URL` env vars.
+
+### Code Formatting
+
+- Go: `gofmt` / `goimports` (post-edit hooks configured)
+- C: `clang-format` using project config at `xce/.clang-format`
+
+### Auth and RBAC
+
+- Auth backend set in `config/auth.yaml`. Supports: configfile, LDAP, OAuth2, PAM.
+- Permissions as `"action:scope"` strings: `"job:view:self"`, `"queue:submit:all"`.
+- RBAC loaded from `config/rbac.yaml`, enforced by `controllers.RequirePermission()` middleware.
+
+## Key Technical Decisions
+
+1. **No Go workspace** — `apiserver` and `xcecli` are independent modules; changes to shared types require updating both.
+2. **In-memory API state** — no database in Go tier; state loaded from YAML at startup via `store.LoadBuiltinData()`.
+3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell.
+4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin.
+5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output.
+6. **Single-writer HA** — controller is always one active primary; standby election via LSF-style priority, not consensus.

.kiro/steering/testing.md

@@ -0,0 +1,89 @@
+# Testing Standards
+
+## Philosophy
+
+- **TDD mandatory**: Write a failing test first, then the minimal implementation. The cycle is Red → Green → Refactor.
+- Test behavior, not implementation details.
+- Tests must run without a live server or external services.
+- A change is not complete until `make test` (and `make test-integration` where applicable) passes.
+
+## Go Tests (apiserver / xcecli)
+
+**Co-located** alongside source: `jobs.go` → `jobs_test.go`. Integration tests use descriptive names: `integration_test.go`, `host_control_test.go`.
+
+**Table-driven** for validation/filter logic:
+
+```go
+tests := []struct {
+    name     string
+    input    SomeRequest
+    wantErr  bool
+}{
+    {"valid request", validReq, false},
+    {"missing field", missingReq, true},
+}
+for _, tt := range tests {
+    t.Run(tt.name, func(t *testing.T) {
+        got, err := preCheckRequest(tt.input)
+        if (err != nil) != tt.wantErr { t.Errorf(...) }
+    })
+}
+```
+
+**State isolation** — reset global store state between tests:
+
+```go
+func resetRBACState() { /* clear in-memory stores */ }
+
+func TestPermissions(t *testing.T) {
+    t.Cleanup(resetRBACState)
+    // ...
+}
+```
+
+**Fixtures**: use `t.TempDir()` for file-system state; never rely on absolute paths or external services.
+
+**Build targets**:
+- `make test` — unit tests only (`CGO_ENABLED=0`)
+- `make test-backend` — CGO-backed backend tests (requires nix dev shell + build dir)
+- `make test-integration` — full regression suite (requires `XCE_BIN` and `SRV_URL` env vars)
+
+## C Tests (controller / worker / scheduler / lib)
+
+**Location**: `testsuite/` directory inside each C component.
+
+**Structure**: one `.c` test program per unit:
+
+```
+src/controller/testsuite/
+    test.c                       ← test runner harness
+    test_ha_cluster.c            ← tests for ha.c
+    test_persist_wal.c           ← WAL-specific tests
+    test_e2e_checkpoint.c        ← end-to-end checkpoint flow
+    test_p1_integration.c        ← P1 HA integration tier
+```
+
+File naming convention: `test_<unit>.c`. E2E tests: `test_e2e_<topic>.c`. Integration tiers: `test_p1_*.c`, `test_p2_*.c`, `test_p3_hardening.c`.
+
+**Test-only symbols** are guarded to prevent production bloat:
+
+```c
+#ifdef XCE_TESTING
+extern void reset_ha_state(ha_state_t *s);
+#endif
+```
+
+**Add a test case in `testsuite/` before modifying any C implementation.**
+
+## Coverage Targets
+
+- Go: 80% minimum on business logic (controllers, services, store)
+- C: every new function in a daemon must have at least one test case in `testsuite/`
+- Integration: all public REST endpoints must have at least one integration test case in `regression/`
+
+## What Not to Mock
+
+- Mock **external dependencies only** (network calls, file I/O that crosses component boundaries)
+- Do **not** mock the system under test
+- In-memory stores (`store/`) are real — use them directly in unit tests
+- CGO bridge (`backend/`) is covered by `make test-backend` — do not stub it in unit tests