lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(libxce-failover): foundation — per-ctx failover state, redirect-hint fields, remove orphaned retry primitive (task 1.1)

Browse source
This commit is contained in:
Rongsong Shen 2026-06-17 00:05:41 +00:00
parent 8c11655e53
commit 387ccfa8dc
17 changed files with 2122 additions and 562 deletions
Download patch file Download diff file Expand all files Collapse all files

150
.kiro/specs/libxce-failover/brief.md Normal file
View file

@ -0,0 +1,150 @@
# Brief: libxce-failover
## Problem
Client applications using libxce do **not** survive controller primary failover. When the
primary controller is killed and a new one is elected, in-flight and subsequent libxce calls
fail with an opaque `Invalid argument` (EINVAL) instead of either (a) retrying transparently
against the new primary or (b) reporting a clear "cluster is not ready" during the election
window.
Reproduction (3-node HA cluster, `set masters localhost:9991,localhost:9992,localhost:9993`):
1. Primary elected; `jobsub` / `joblist` work.
2. Kill the primary (`ctrl-1`).
3. Before a new primary is elected, `joblist` should report **cluster not ready**.
4. After a new primary is elected, all commands should transparently route to the new
primary with no error.
Observed instead: `joblist` keeps returning stale cached output, then `xce_joblist: Invalid
argument` indefinitely — the client never fails over.
## Current State
- The requirement already exists: **`ha-config-bootstrap` Requirement 7 — "Client-Side
Transparent Retry for Transient Controller Errors"** (acceptance criteria 7.1–7.6), and its
tasks 10 / 10.1 / 10.2 are marked complete.
- Only the **standalone primitive** was delivered:
- `xce_request_with_retry()` is implemented at `src/lib/request.c:1956` and unit-tested in
`src/lib/testsuite/test_libxce_retry.c` (NOT_READY backoff, STANDBY+redirect reconnect,
STANDBY-without-redirect fallback, non-retriable surface-immediately, max-retries,
50-client concurrency). `xce_retry_config_t` defaults: `max_retries=20`,
`initial_retry_ms=50`, `max_retry_ms=5000`, `backoff_multiplier=2.0`.
- **Gap 1 — wiring:** every production API path bypasses the wrapper. `xce_joblist` →
`xce_get_req` (`src/lib/request.c:1373`) → plain `xce_request()` (`request.c:1385`). The
same is true for ~18 other `xce_*_req` wrappers — none call `xce_request_with_retry()`.
- **Gap 2 — multi-master discovery:** the retry primitive operates on a single `endpoint_t`
and only follows an explicit `redirect_host/redirect_port`. A *killed* primary cannot
redirect, so the client must **rotate through its configured master list** (9991/9992/9993)
to discover the new primary. That orchestration does not exist; the client connects to
`masters[0]` once in `xcecli_start()` (`src/lib/client.c:122`) and never re-selects.
- Connections are long-lived and cached in `api_ctx_t->ep`; there is no dead-primary
detection or automatic reconnect at the API level.
## Desired Outcome
Failover is transparent to client applications:
- Short failovers are hidden: libxce retries across the configured masters with backoff and
succeeds against the new primary once elected.
- If the election outlasts the retry budget, the call returns a clear `E_XCE_NOT_READY`
("cluster not ready") rather than `EINVAL`.
- Both read and write requests fail over (writes rely on existing server-side commit-barrier +
idempotency guards to dedup re-sends).
- No per-daemon ad-hoc retry loops — the behavior lives entirely in libxce.
## Approach
**Central chokepoint (Approach A).** Make failover the default behavior at the single point
all wrappers already funnel through:
1. Give the request layer access to the cluster/master list + a "current primary" index
(design choice: a back-reference on `endpoint_t`, or a small `xce_cluster_t` the endpoint
references — resolved in design phase).
2. Upgrade `xce_request_with_retry()` to, in addition to its existing redirect/backoff cases:
- rotate through the master list on connection failure (`ECONNREFUSED`/`ETIMEDOUT`/RDHUP) or
`STANDBY`-without-redirect, reconnecting to each candidate to find the new primary;
- apply the bounded backoff budget and, on expiry, surface `E_XCE_NOT_READY`.
3. Route the production `xce_*_req` wrappers through the retry/failover path (e.g. make
`xce_request()` the failover wrapper over an inner `xce_request_raw()`), wiring every API
at once.
Chosen because it is the only approach that covers the entire client surface at one
integration point, reuses the already-tested primitive, treats reads and writes uniformly, and
keeps the requirement out of caller code — matching "transparent to the client application."
## Scope
- **In:**
- Wire all production libxce request paths through transparent retry/failover.
- Multi-master rotation to discover the new primary after the connected one dies.
- Bounded-budget semantics: transparent retry, then `E_XCE_NOT_READY` ("cluster not ready").
- Failover for both read and write requests (rely on server idempotency for writes).
- Tests: extend `test_libxce_retry.c` for master-list rotation + budget→NOT_READY; a 3-node
live failover integration test (kill primary, assert NOT_READY window then transparent
recovery).
- **Out:**
- Server-side election, WAL, commit barrier, idempotency guards (already owned by the HA
server specs; this spec consumes them).
- Changing the `xce_test` REPL behavior beyond what surfaces the new error semantics.
- Per-request idempotency/retry opt-in flags (writes retry by default per decision).
## Boundary Candidates
- Master-list + current-primary selection state and rotation policy.
- The retry/failover wrapper (backoff budget, error classification, NOT_READY surfacing).
- Integration seam: how production `xce_*_req` wrappers reach the failover path.
- Error-surface mapping: connection/EINVAL vs. `E_XCE_NOT_READY` vs. `E_XCE_STANDBY` vs.
non-retriable (auth/perm/invalid/noent).
## Out of Boundary
- Any controller/server-side change (election timing, redirect content, idempotency).
- New transport/probe opcodes unless design shows rotation cannot work with existing replies.
## Upstream / Downstream
- **Upstream:** `ha-config-bootstrap` (Req 7 retry primitive + `xce_retry_config_t` + error
codes `E_XCE_NOT_READY` / `E_XCE_STANDBY` + `redirect_host/port` reply fields);
`ha-commit-barrier-rollout` (server idempotency guards that make write retries safe);
`ha-single-leader-role` / election specs (define when a standby/primary answers).
- **Downstream:** worker, scheduler, and apiserver clients — all gain transparent failover and
can drop any ad-hoc retry handling; the `/xce-deploy-ha` live test harness.
## Existing Spec Touchpoints
- **Extends:** consumes and completes the intent of `ha-config-bootstrap` Req 7 (the primitive
was delivered but never integrated); this spec owns the integration + multi-master failover.
- **Adjacent:** `ha-repl-hardening`, `ha-convergence-recovery` (server-side failover behavior);
`worker-rejoin-resync` (worker-side rejoin, distinct from client-library failover).
## Protocol Correction (2026-06-16, requirements phase)
The discovery notes above assumed the controller signals non-primary via `E_XCE_STANDBY` with
`redirect_host/redirect_port` fields. Code review found the real taxonomy (`errnox.h:22-25`):
`E_XCE_REDIRECT` (standby knows a primary exists; `join.c:260-268`), `E_XCE_NOMASTER` (standby,
no primary known yet; `join.c:270-278`), `E_XCE_NOT_READY` (INIT/ELECTING), `E_XCE_STANDBY`
(primary-only op on a standby). The controller's redirect path sends `E_XCE_REDIRECT` with
`data_which = nodata` — **status only, no address** — so the existing `xce_request_with_retry()`
condition (`E_XCE_STANDBY` + populated redirect fields, `request.c:2010`) never matches a real
controller. Decisions (confirmed): classify on the actual statuses — `E_XCE_REDIRECT` →
fast-rotate the master list to find the existing primary (use an address if the reply carries
one, else rotate); `E_XCE_NOMASTER`/`E_XCE_NOT_READY` → backoff → cluster-not-ready;
`E_XCE_STANDBY` → rotate. The existing retry unit tests (keyed on the non-emitted shape) are
corrected to the real statuses. Tracked as bug-608.
**Scope expansion (confirmed):** the redirect address is absent today only because `join.c`
never fills the reply's `redirect_host/redirect_port` fields — those fields already exist in the
wire schema (`controller.capnp:568-569`) and the reply struct (`typemap.h:225-226`), and the
controller already knows the address (`join.c:263`). One small controller change to populate them
is now **in scope** (Requirement 8), making redirect-based failover deterministic (direct connect
to the primary, O(1)); client-side master-list rotation remains as a defensive fallback for an
older controller that omits the address.
## Constraints
- C library (libxce); follow project error-handling (`errnox`, early-return) and async-I/O
rules — `xce_request`/`xce_reply` non-blocking; never block the iomgr thread.
- TDD mandatory: failing tests in `src/lib/testsuite/` before implementation.
- Touches HA/failover behavior → full Kiro workflow (requirements → design → tasks), not the
quick path, per CLAUDE.md.
- Reuse the existing `xce_request_with_retry()` primitive and its config struct; do not fork a
second retry implementation.

540
.kiro/specs/libxce-failover/design.md Normal file
View file

@ -0,0 +1,540 @@
# Design Document — libxce-failover
## Overview
**Purpose**: This feature delivers transparent controller-primary failover to every client
application built on libxce (worker, scheduler, apiserver via cgo, and the `xce_test` harness),
so that a controller failover is hidden when short and reported clearly when not.
**Users**: libxce callers issue normal requests (`xce_joblist`, `xce_jobsub`, …). When the
connected primary dies, libxce locates the newly elected primary and completes the request there,
or — if no primary exists yet — returns a distinct cluster-not-ready error instead of an opaque
`EINVAL`. No caller code changes.
**Impact**: Today the client connects to one master (`client.c:122`), routes every request through
the fire-and-forget `xce_request()`, and has no reply-status-driven retry on the production path
(`xce_request_with_retry()` only retries in test-hook mode). This design places failover at the
**`client.c` (`xcecli_*`) layer** — the universal client surface that every consumer goes through
(worker, scheduler, apiserver-via-cgo, `xce_test`) and the only layer whose context (`api_ctx_t`)
co-locates the master list (`conf`), the endpoint (`ep`), and the reply context (`ctx`). Each
`xcecli_*` op already calls its internal `xce_*` op, which **returns the controller status**; the
failover loop wraps that call, classifies the status, and reconnects/rotates over `api_ctx.conf`.
It adds master-list rotation, redirect-address following, and a small controller change so redirect
replies carry the primary address. `xceapi.c` ops are unchanged.
### Goals
- Route all libxce request/reply round trips through one transparent failover path (1.1, 1.3).
- Discover the new primary via redirect address, falling back to master-list rotation (2.x, 8.x).
- Hide short failovers via bounded backoff; surface a distinct cluster-not-ready on exhaustion (3.x).
- Fail over reads and writes uniformly, relying on controller idempotency for writes (5.x).
- Reuse the existing retry config/defaults; stagger retries with jitter (6.x).
### Non-Goals
- Controller-side election, WAL, commit barrier, write-idempotency logic (consumed, not changed).
- Per-daemon adoption (master-list wiring, liveness-safe blocking) — `component-failover-adoption`.
- New transport opcodes or new reply fields (redirect fields already exist).
## Boundary Commitments
### This Spec Owns
- The client-side failover loop: classify status → reconnect/rotate/backoff → resume, including the
cluster-not-ready terminal mapping, for **reply-bearing** `xcecli_*` ops.
- Client-side master-list cursor and current-primary stickiness within a context.
- Capturing the redirect address from a reply as a best-effort hint (no correctness dependency).
- A **reconnect/rejoin hook**: a consumer-registered callback the failover machinery invokes after
it reconnects to a new primary and *before* it resumes the interrupted op. libxce owns the hook
mechanism, its ordering, and invoking it; it does **not** own the callback's content.
- A per-ctx concurrency lock so `xcecli_*` failover is safe when one ctx is shared across threads.
- The single controller change that populates `redirect_host/port` on `E_XCE_REDIRECT` replies.
- Realignment of the libxce retry unit tests to the real controller status taxonomy.
### Out of Boundary
- How the controller decides primary/standby/redirect/no-primary (election, `ha_should_redirect`).
- The **content** of the rejoin callback: the join/state-exchange a daemon performs on reconnect
(owned by `component-failover-adoption`) and the worker's running-jobs re-sync (owned by
`worker-rejoin-resync`). This spec provides and invokes the hook; the daemons fill it in.
- Redelivery of fire-and-forget notifications lost across a failover (also `worker-rejoin-resync`).
- Worker/scheduler/apiserver adoption and their threading/liveness handling of bounded blocking.
- Any reply payload beyond the already-defined `redirect_host/redirect_port`.
### Allowed Dependencies
- Existing controller status codes (`errnox.h:22-25`) and redirect reply fields (`typemap.h:225`).
- `endpoint_connect()` / endpoint I/O (`netio.*`), request encode/enqueue (`request.c`), and the
context wait channel (`cbuf`, `xcectx.c`).
- `xce_retry_config_t` + `XCE_RETRY_DEFAULTS` (request.h) and the cluster/peer config (`cluster.*`).
### Revalidation Triggers
- Change to the reply `status`/`redirect_*` contract → re-check client classification + controller.
- Change to the async reply-dispatch model (`handle_reply` → `cbuf` signalling) → re-check round trip.
- Change to `api_ctx_t`/endpoint ownership → re-check reconnect-reuses-same-endpoint assumption.
- New non-primary status code added to the controller → extend client classification.
- Change to the reconnect/rejoin hook signature or its invocation ordering → re-check daemon rejoin
registration (`component-failover-adoption`) and worker resync (`worker-rejoin-resync`).
## Architecture
### Existing Architecture Analysis
- **Layering — `xcecli_*` is the universal surface.** Every consumer calls `client.c`'s `xcecli_*`
ops, never `xceapi.c`'s `xce_*` directly: worker (`controller_connector.c:184,272`), scheduler
(`scheduler.c:457,463`), apiserver via cgo (`clib.go:34,36`), `xce_test`. Each `xcecli_*` is a thin
wrapper that calls its `xce_*` counterpart, which **returns the controller status** as its rc.
- **Only `api_ctx_t` reaches the master list.** `api_ctx_t = { conf (masters), ep, ctx }`
(client.c:23). The inner `xce_ctx_t` (xceapi) holds `ep`/`wbuf`/`state` but **no** `conf`
(xcectx.h) — so failover belongs at the client layer, where the master list is already in scope.
- **`xce_*` fail fast; `xcecli_*` retries.** The async send/wait stays inside `xce_*` unchanged: a
single attempt that returns the controller status (or transport error). The `xcecli_*` wrapper
classifies that rc and, on a retriable status, reconnects/rotates `api_ctx.ep` over `api_ctx.conf`
and re-invokes `xce_*`. No async internals, thunks, or master-list plumbing are introduced.
- **Reconnect reuses the endpoint.** `endpoint_connect(ep,host,port)` re-points the same
`endpoint_t`, preserving the `msg_handler` wiring that routes replies to `ctx->wbuf`.
- **Stateful consumers need a rejoin on reconnect.** `xce_test` is a pure client — reconnect +
resend suffices. But the apiserver, worker, and scheduler hold a *session* with the primary
established via `xcecli_join` (role + attrs); a freshly elected primary has no live session for a
reconnecting daemon until it re-joins, and a worker must additionally re-sync its running-jobs
state. So failover for these consumers is **reconnect → rejoin/state-exchange → resume**, not just
reconnect → resend. libxce models this with a consumer-registered reconnect hook; the hook content
lives in `component-failover-adoption` (join) and `worker-rejoin-resync` (job state).
- **Two op shapes.** Reply-bearing ops (`xce_joblist`/`xce_jobsubmit`/`xce_hello`/…) do
`xce_*_req` + `cbuf_get` → return a controller status the loop can classify. Fire-and-forget ops
(`xce_notify*`, heartbeat) do `xce_*_req` only (no `cbuf_get`, xceapi.c:282) → return the enqueue
result, never a controller status. Failover classification applies only to the former.
- **Patterns preserved**: `xce_*` ops, `xce_*_req` enqueue wrappers, and `xce_request`/`xce_reply`
are all unchanged; failover is additive in `xcecli_*`. Trade-off: ~33 `xcecli_*` wrappers each gain
a small retry loop (more lines), but the policy lives in one shared decision helper (fewer moving
parts) — an explicit simplicity-over-LOC choice.
### Architecture Pattern & Boundary Map
```mermaid
graph LR
Caller[consumer worker scheduler apiserver xce_test] --> Cli[xcecli op in client.c]
Cli --> Loop[do while retry loop]
Loop --> Decide[xcecli_retry_next decision helper]
Loop --> Xce[xce op in xceapi one shot]
Xce --> Ctrl[controller]
Ctrl --> Reply[handle_reply]
Reply --> Capture[capture redirect hint into ctx]
Decide --> Rotate[reconnect or rotate over api_ctx conf and ep]
Decide --> Backoff[jittered backoff]
```
- **Selected pattern**: a thin retry loop in each `xcecli_*` wrapper, driven by one shared decision
helper (`xcecli_retry_next`). Rationale: `client.c` is the universal surface AND the only layer
whose `api_ctx_t` co-locates master list + endpoint + ctx; `xce_*` already returns the status.
- **Dependency direction**: `types → cluster/config → endpoint(netio) → xceapi(xce_*) → client(xcecli_*)`.
Each layer imports only leftward. Failover logic is added at the `client` layer (rightmost),
depending on `xce_*` and `cluster/config`; `xceapi` does not depend on it.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Client wrappers | C (gnu17), libxce `client.c` | `xcecli_*` retry loop + decision helper + rotate | Reuses `xce_*`, `endpoint_connect`, `cluster` |
| Client API | C (gnu17), libxce `xceapi.c` | **Unchanged** — one-shot `xce_*` returns status | No edits |
| Controller | C (gnu17) | Populate redirect address on `E_XCE_REDIRECT` | ~2 lines in `join.c`; fields already in schema |
| Tests | GoogleTest (testsuite) | Realign retry tests to real status taxonomy | Test the `xcecli_*` retry path |
## File Structure Plan
### New Files
```
src/lib/client_failover.c # xcecli_retry_next decision helper, StatusClassifier, rotate/reconnect over api_ctx
src/lib/client_internal.h # private: api_ctx_t + xce_failover_state_t shared by client.c and client_failover.c
```
> `api_ctx_t` is currently file-local to `client.c`; move its definition to `client_internal.h` so the
> failover helper can operate on it without exposing it in the public API.
### Modified Files
- `src/lib/client.c` — wrap each **reply-bearing** `xcecli_*` op body in a `do { rc =
xce_<op>(...); } while (xcecli_retry_next(ac,&rc,is_write,&st));` loop (per OpCategory);
fire-and-forget ops get the transport-reconnect-only variant; add `xce_failover_state_t` to
`api_ctx_t` (lock + cursor + retry config + rejoin hook); add `xcecli_set_reconnect_handler()`;
init + take `failover.lock` per op; init in `xcecli_start()`; keep initial connect to `masters[0]`.
- `src/include/client.h` — declare `xcecli_set_reconnect_handler()` + `xcecli_reconnect_cb`.
- `src/lib/xcectx.c` — `handle_reply()` copies the redirect-hint address (`reply.redirect_host/port`,
when present) into `xce_ctx_t` before the reply is freed; the existing error path (`XCE_CTX_ERR` +
`wbuf` error inject, xcectx.c:137-154) is the wake hook reused on reconnect.
- `src/include/xcectx.h` — add the fixed-size redirect-hint `host/port` fields to `xce_ctx_t`.
- `src/include/request.h` / `src/lib/request.c` — keep `xce_retry_config_t` + `XCE_RETRY_DEFAULTS`;
**remove** the orphaned no-op `xce_request_with_retry()` (superseded by the `xcecli_*` loop).
- `src/lib/xceapi.c` — **unchanged** (one-shot `xce_*` ops already return the controller status).
- `src/controller/join.c` — populate `reply.redirect_host = c->ha.primary_host` /
`reply.redirect_port = c->ha.primary_port` on the `E_XCE_REDIRECT` path (audit for other sites).
- `src/lib/testsuite/test_libxce_retry.c` — retarget from the retired primitive to the `xcecli_*`
retry path + `StatusClassifier`; cover `E_XCE_REDIRECT`/`NOMASTER`/`NOT_READY`/`STANDBY`,
master rotation, and budget→not-ready.
## System Flows
### Failover loop (per `xcecli_*` op)
```mermaid
sequenceDiagram
participant C as Caller
participant F as xcecli op do while loop
participant E as Endpoint
participant K as Controller
C->>F: xcecli op
loop until success, terminal, or budget exhausted
F->>E: xce op one shot
E->>K: request
K-->>E: reply status plus optional redirect addr
E-->>F: xce op returns status
alt status success
F-->>C: ok
else redirect or standby or transport error
F->>E: rotate (hint address first if set, else next master) then invoke rejoin hook
else no-primary or not-ready
F->>F: backoff jittered then rotate
else non-retriable
F-->>C: surface error
end
end
F-->>C: cluster-not-ready on budget exhaustion
```
After any reconnect to a new endpoint, the failover machinery invokes the consumer-registered
reconnect/rejoin hook (if set) **before** re-sending the interrupted op, so a stateful daemon
re-establishes its session (and a worker re-syncs job state) against the new primary. Pure clients
(`xce_test`) register no hook and skip straight to re-send.
Key decisions: redirect/standby/transport rotate **without** the no-primary backoff delay (4.2);
only no-primary/not-ready consume the backoff budget (3.1, 4.1); on success the served master
becomes the sticky current primary (2.4).
### Status classification
```mermaid
stateDiagram-v2
[*] --> Classify
Classify --> Success: status==0 or DUP on re-sent write
Classify --> RetryNow: REDIRECT or STANDBY or transport error
Classify --> RetryBackoff: NOMASTER or NOT_READY
Classify --> Terminal: AUTH PERM INVALID NOENT
RetryNow --> Rotate
RetryBackoff --> Rotate
Rotate --> Classify: hint first else next master
Success --> [*]
Terminal --> [*]
```
## Requirements Traceability
| Requirement | Summary | Components | Flows |
|-------------|---------|------------|-------|
| 1.1, 1.3 | All ops through failover; preserve API | XcecliRetryLoop (all `xcecli_*`) | Failover loop |
| 1.2 | Complete next request against new primary | XcecliRetryLoop, EndpointRotator | Failover loop |
| 2.1 | Unreachable → try other masters | EndpointRotator | Classification |
| 2.2, 8.3, 8.4 | Redirect addr if present else rotate | RedirectCapture, EndpointRotator | Classification |
| 2.3, 4.3 | Standby → rotate | StatusClassifier, EndpointRotator | Classification |
| 2.4 | Sticky current primary | FailoverState | Failover loop |
| 2.5 | Redirect/standby/no-primary ≠ success | StatusClassifier | Classification |
| 3.1, 4.1 | No-primary → backoff to budget | XcecliRetryLoop, FailoverState | Failover loop |
| 3.2 | Success before budget | XcecliRetryLoop | Failover loop |
| 3.3, 3.4 | Budget exhausted → distinct cluster-not-ready | XcecliRetryLoop | Failover loop |
| 4.2 | Redirect → prompt switch, no backoff | EndpointRotator | Classification |
| 4.4 | Transport → reconnect/rotate | EndpointRotator, ReplyWaitWake | Classification |
| 4.5 | Non-retriable → surface | StatusClassifier | Classification |
| 1.2 | Re-establish session against new primary | ReconnectRejoinHook | Failover loop |
| 5.1, 5.3 | Reads+writes; no silent drop (reply-bearing) | XcecliRetryLoop, OpCategory | Failover loop |
| 5.2 | Write retry not double-applied | StatusClassifier (DUP→success) + controller idempotency | — |
| 6.1, 6.2, 6.3 | Configurable params, defaults, jitter | FailoverState, XcecliRetryLoop | Failover loop |
| 7.1, 7.2 | Log failover target + exhaustion | XcecliRetryLoop | — |
| 8.1, 8.2 | Controller populates redirect addr | ControllerRedirectAddress | — |
## Components and Interfaces
| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| XcecliRetryLoop | client (`client.c`) | `do/while` wrapper around each reply-bearing `xce_*` call | 1.1,1.2,3.x,5.x,7.x | xcecli_retry_next(P0) | Service |
| xcecli_retry_next | client (`client_failover.c`) | Decision helper: classify, rotate, rejoin, backoff, budget | 1.2,3.x,4.x,7.x | EndpointRotator(P0), StatusClassifier(P0), ReconnectRejoinHook(P0), FailoverState(P0) | Service |
| ReconnectRejoinHook | client (`client.c`) | Invoke consumer callback after reconnect, before resume | 1.2 | api_ctx_t(P0) | Service, State |
| OpCategory | client (`client.c`) | Per-op shape: reply-bearing / write / fire-and-forget | 1.1,5.1,5.2 | — | State |
| EndpointRotator | client (`client_failover.c`) | Reconnect to addr or next master over api_ctx | 2.1,2.2,2.3,4.2,4.4 | endpoint_connect(P0), FailoverState(P0) | Service |
| StatusClassifier | client (`client_failover.c`) | Map status → action | 2.5,4.1,4.2,4.3,4.5,5.2 | errnox codes(P0) | Service |
| FailoverState | client (`api_ctx_t`) | Master cursor, retry cfg, sticky primary | 2.4,6.1 | xce_client_config_t(P0) | State |
| RedirectCapture | ctx (`xcectx.c`) | Copy redirect-hint address into ctx (if present) | 2.2,8.3 | xce_reply_t(P0) | State |
| ReplyWaitWake | ctx (`xcectx.c`) | Release cbuf_get on endpoint error | 4.4 | endpoint err_handler(P0) | Service |
| ControllerRedirectAddress | controller | Fill redirect_host/port | 8.1,8.2 | ha_state primary_host/port(P0) | State |
### client (libxce)
#### XcecliRetryLoop — the wrapper pattern in each `xcecli_*`
| Field | Detail |
|-------|--------|
| Intent | Turn each one-shot `xce_*` call into a transparent failover loop with no change to `xce_*` |
| Requirements | 1.1, 1.2, 3.1, 3.2, 3.3, 3.4, 5.1, 5.3, 7.1, 7.2 |
**Responsibilities & Constraints**
- Each `xcecli_*` op runs its existing `xce_*` call inside a `do/while` driven by `xcecli_retry_next`.
- `xce_*` ops are unchanged: each is a single attempt that returns the controller status (fail fast).
- Never drops a request silently (5.3): the loop returns the op's success value or a distinct error.
**Contracts**: Service [x]
```c
/* Pattern applied to every xcecli_* op (joblist shown; is_write=false for reads). */
int xcecli_joblist(xcecli_ctx_t c, xce_job_t ***out) {
api_ctx_t *ac = (api_ctx_t *)c;
xcecli_retry_t st = XCECLI_RETRY_INIT(ac); /* attempt=0, delay/seed from ac->failover.retry */
int rc;
do {
rc = xce_joblist(ac->ctx, out); /* unchanged one-shot xceapi op */
} while (xcecli_retry_next(ac, &rc, /*is_write=*/false, &st));
return rc; /* >=0 ok; -E_XCE_NOT_READY = cluster not ready */
}
```
- Postconditions: `>=0` success (sticky primary updated, 2.4); `-E_XCE_NOT_READY` on no-primary
budget exhaustion (3.3); terminal errors surfaced as-is (4.5). For writes, pass `is_write=true`.
#### xcecli_retry_next — decision helper
| Field | Detail |
|-------|--------|
| Intent | Decide whether to retry after an `xce_*` returned `rc`; perform reconnect/rotate/backoff |
| Requirements | 1.2, 3.1, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 5.2, 7.1, 7.2 |
**Responsibilities & Constraints**
- Owns the retry budget, backoff progression, sticky-primary update, and cluster-not-ready mapping.
- After a transport error the existing path left `ctx->state = XCE_CTX_ERR`; on a successful reconnect
this helper restores `XCE_CTX_RUN` so the next `xce_*` passes its guard (CI2).
- Drains any stale entry from `ctx->wbuf` after reconnect, before the next attempt (CI1); reconnect
closes the old fd first, so the dead connection yields no further replies.
**Contracts**: Service [x]
```c
typedef struct { int attempt; int delay_ms; unsigned int rand_seed; } xcecli_retry_t;
/* Returns true → caller re-invokes its xce_* op; false → stop, *rc is final.
* On DUP write replay it sets *rc = 0 (success) and returns false (5.2). */
bool xcecli_retry_next(api_ctx_t *ac, int *rc, bool is_write, xcecli_retry_t *st);
```
- Decisions: SUCCESS/TERMINAL → return false; RETRY_NOW → `EndpointRotator` (hint-first) then true,
no backoff (4.2); RETRY_BACKOFF → jittered sleep then `EndpointRotator` then true (4.1); attempts
beyond `retry.max_retries` while no-primary → set `*rc = -E_XCE_NOT_READY`, return false (3.3, 3.4).
- Validation: INFO log target on rotation (7.1); WARN log attempt count on exhaustion (7.2).
#### EndpointRotator
| Field | Detail |
|-------|--------|
| Intent | Re-point `ac->ep` to a redirect address or the next master in `ac->conf` |
| Requirements | 2.1, 2.2, 2.3, 4.2, 4.4, 8.4 |
**Responsibilities & Constraints**
- On retry, if a redirect-address **hint** was captured, try it first (then clear it); otherwise
advance the master cursor and reconnect to the next configured master (8.3, 8.4). A failed hint
just falls through to rotation on the next retry — correctness never depends on it.
- Reuse the same `endpoint_t` (reply routing to `ctx->wbuf` preserved); close stale fd/ssl first.
**Contracts**: Service [x]
```c
/* Reconnect ac->ep to the redirect-hint address if one is set (consumes it), else advance to the
* next configured master in ac->conf and reconnect. Returns 0 or -errno. */
int xce_failover_rotate(api_ctx_t *ac);
```
#### StatusClassifier
| Field | Detail |
|-------|--------|
| Intent | Map a reply status / transport rc to a failover action |
| Requirements | 2.5, 4.1, 4.2, 4.3, 4.5, 5.2 |
**Contracts**: Service [x]
```c
typedef enum {
XCE_FO_SUCCESS, /* status == 0, or idempotent-replay success (DUP on a re-sent write) */
XCE_FO_RETRY_NOW, /* E_XCE_REDIRECT / E_XCE_STANDBY / transport error — rotate, no backoff */
XCE_FO_RETRY_BACKOFF, /* E_XCE_NOMASTER / E_XCE_NOT_READY — rotate after jittered backoff */
XCE_FO_TERMINAL /* AUTH/PERM/INVALID/NOENT and other non-retriable — surface */
} xce_fo_action_t;
/* attempt == 0 for the first send; >0 for a re-send after failover.
* is_write distinguishes mutating ops (job submit) from reads.
* The redirect ADDRESS is NOT a classifier concern — it is a hint consumed by EndpointRotator. */
xce_fo_action_t xce_failover_classify(int status, bool is_write, int attempt);
```
- Invariants: `E_XCE_REDIRECT`/`STANDBY`/`NOMASTER`/`NOT_READY` are never reported as success (2.5).
- **Idempotent-replay rule (5.2):** `-E_XCE_DUP_JOB` / `-EEXIST` on a **re-sent write** (`is_write &&
attempt > 0`) maps to `XCE_FO_SUCCESS` — the controller's client-supplied-`jobid` replay guard
(`job.c:56,95`) means "already applied" = our write succeeded. On `attempt == 0` the same status
is a genuine duplicate and maps to `XCE_FO_TERMINAL` (surfaced unchanged).
#### OpCategory — per-op shape (CI1)
| Field | Detail |
|-------|--------|
| Intent | Classify each `xcecli_*` op so its wrapper applies the right failover behavior |
| Requirements | 1.1, 5.1, 5.2 |
Three categories drive the wrapper generated for each of the ~33 ops:
| Category | Ops (examples) | `xce_*` shape | Failover behavior |
|----------|----------------|---------------|-------------------|
| Reply-read | `joblist`, `reslist`, `workerlist`, `clusterconf`, `resval`, `hello`, `nop` | `*_req` + `cbuf_get` → status | Full failover; `is_write=false` |
| Reply-write | `jobsubmit`, `jobctrl`, `queuectrl`, `hostctrl`, `update_resvals`, `runtask`, `join` | `*_req` + `cbuf_get` → status | Full failover; `is_write=true` (DUP-replay→success) |
| Fire-and-forget | `notify_*` (jobstart/finish/fail/stop/chkpt), `heartbeat`, `wakeup` | `*_req` only, **no status** | Best-effort: reconnect on a known-dead connection only; **no** standby/redirect detection, **no** no-primary backoff. Redelivery deferred to `worker-rejoin-resync` |
- The exact per-op assignment is enumerated during task generation; `join` is reply-write and is the
op that triggers `E_XCE_REDIRECT` at reconnect time (it carries the daemon's role/attrs).
#### ReconnectRejoinHook
| Field | Detail |
|-------|--------|
| Intent | Let a stateful consumer re-establish its session on the new primary after a reconnect, before the interrupted op resumes |
| Requirements | 1.2 |
**Responsibilities & Constraints**
- libxce owns the hook *mechanism* and *ordering* (reconnect → hook → resume); the callback *content*
(join handshake, worker job re-sync) belongs to `component-failover-adoption` / `worker-rejoin-resync`.
- Invoked once per successful reconnect to a *new* endpoint, before the next `xce_*` attempt. If the
hook fails (e.g. the new primary rejects the join), the failover loop treats it like a failed
attempt (rotate/backoff per the returned status). Pure clients register no hook (no-op path).
**Contracts**: Service [x] / State [x]
```c
/* Registered via the client API; cb runs on the calling thread after reconnect.
* Return 0 to proceed with the interrupted op; <0 to treat the rejoin as failed. */
typedef int (*xcecli_reconnect_cb)(xcecli_ctx_t ctx, void *arg);
int xcecli_set_reconnect_handler(xcecli_ctx_t ctx, xcecli_reconnect_cb cb, void *arg);
```
### client / ctx (libxce)
#### FailoverState (on `api_ctx_t`)
**Contracts**: State [x]
```c
/* On api_ctx_t — persists across calls (sticky primary + tunables). */
typedef struct {
mylock_t lock; /* serializes xcecli_* per ctx (CI2 — shared-ctx safety) */
int cur_master; /* index into conf->masters[] (sticky primary) */
xce_retry_config_t retry; /* defaults from XCE_RETRY_DEFAULTS unless overridden */
xcecli_reconnect_cb rejoin_cb; /* consumer rejoin hook; NULL for pure clients */
void *rejoin_arg;
} xce_failover_state_t;
/* On xce_ctx_t — written by handle_reply (which only sees xce_ctx_t), read by the rotator via ac->ctx. */
/* char redirect_hint_host[XCE_MAXHOSTNAME_LEN]; hint from last redirect reply; "" if none */
/* uint16_t redirect_hint_port; */
```
- State model: `cur_master` updated on a successful served-as-primary reply (2.4); retry config from
`XCE_RETRY_DEFAULTS` unless overridden (6.1, 6.2). The redirect hint is read by the rotator on the
next retry and cleared on consumption; it is purely advisory.
- **Concurrency (CI2):** the apiserver shares one ctx across goroutines (a heartbeat goroutine plus
request handlers, `clib.go:182`). Each `xcecli_*` op takes `failover.lock` for the duration of its
`do/while` loop, so reconnect/rotate (which closes `ac->ep` and drains `wbuf`) cannot race another
in-flight call on the same ctx. The single-slot `wbuf` already implies one in-flight request per
ctx; the lock makes that explicit and covers the reconnect window.
#### RedirectCapture (in `handle_reply`)
**Contracts**: State [x]
- On a redirect reply that carries an address, copy `reply.redirect_host` (capnp-owned, freed with
the reply — see request.c:243-250 hazard) + `redirect_port` into the fixed-size `xce_ctx_t` hint
fields before the reply is freed; leave the hint empty otherwise. Because the hint is advisory, the
copy needs no sequence correlation or lifetime guarantees beyond "copy now, use-or-discard next
retry." The status continues through the existing `cbuf_tryput(&ctx->wbuf,&status)` path
(xcectx.c:303), so `xceapi.c` is unchanged.
#### ReplyWaitWake (endpoint error path)
**Contracts**: Service [x]
- On endpoint error/RDHUP for `ctx->ep`, release a pending `cbuf_get` with a transport error
(`-ECONNRESET`/`-ETIMEDOUT`) so the `xce_*` call returns and the loop rotates instead of hanging
(4.4, F7). The existing error path already injects an error into `wbuf` and sets `ctx->state =
XCE_CTX_ERR` (xcectx.c:148-154) — this component reuses it.
- **State reset (CI2):** that error path leaves `ctx->state = XCE_CTX_ERR`, which the per-op guard
rejects as `-EINVAL` (xceapi.c:14,38,…) — the reported bug. `xcecli_retry_next` restores
`ctx->state = XCE_CTX_RUN` after a successful reconnect, before the next `xce_*` attempt, so the
guard (re-checked per attempt) passes. **CI1 drain:** the helper also drains any stale `wbuf` entry
after reconnect so a late reply from the dead connection is never consumed by the next attempt.
### controller
#### ControllerRedirectAddress (`join.c`)
**Contracts**: State [x]
- On the `E_XCE_REDIRECT` path, set `reply.redirect_host = c->ha.primary_host` and
`reply.redirect_port = c->ha.primary_port` (8.1). When the primary is unknown, keep returning
`E_XCE_NOMASTER` rather than an addressless redirect (8.2 — already the case at join.c:270).
- Audit during implementation: confirm `join.c` is the sole `E_XCE_REDIRECT` emitter; apply the
same populate to any other site found.
## Error Handling
### Error Strategy
Classify every round-trip outcome (StatusClassifier) into success / rotate-now / backoff / terminal,
and bound the loop by the retry budget. Repair the context on transport failure rather than leaving
it in a non-RUN state (F7) so failover is repeatable.
**Context state model (CI2):** the existing transport-error path sets `ctx->state = XCE_CTX_ERR`
(xcectx.c:148); the per-op `XCE_CTX_RUN` guard then rejects with `-EINVAL` (xceapi.c:14,…) — the
reported bug. `xcecli_retry_next` owns the transition back: after a successful reconnect it sets
`ctx->state = XCE_CTX_RUN` before the next `xce_*` attempt, so the per-op guard (re-checked each
attempt) passes. The guard otherwise stays exactly as today — `xceapi.c` is unchanged.
**Stale-reply avoidance (CI1):** `ctx->wbuf` is a single-slot, status-only channel
(`cbuf_init(...,1,sizeof(int))`, xcectx.c:185). Reconnect closes the old fd **first**, so the dead
connection produces no further replies; `xcecli_retry_next` then drains any entry already sitting in
`wbuf` before the next attempt. This drain-after-close is sufficient — no per-reply sequence
correlation is needed (a deliberate simplification).
### Error Categories and Responses
- **Transport (refused/timeout/reset)**: reconnect/rotate, retry; wake any pending wait (4.4).
- **No primary (NOMASTER/NOT_READY)**: jittered backoff; on budget exhaustion return the distinct
`-E_XCE_NOT_READY` cluster-not-ready code (3.3, 3.4) — never `-EINVAL`.
- **Redirect/standby**: rotate (address-first) without backoff (4.2, 4.3).
- **Terminal (AUTH/PERM/INVALID/NOENT)**: surface immediately, unchanged (4.5).
### Monitoring
INFO log on each rotation with the target endpoint (7.1); WARN log on budget exhaustion with the
attempt count and last status (7.2). Reuse existing libxce logging.
## Performance & Scalability
Rejoin-on-failover cost is **accepted as-is** (decision 2026-06-16): failover is a rare, exceptional
event, so the cost is not on any hot path, and *some* handshake is unavoidable — a freshly elected
primary has no live connection to a reconnecting daemon until it re-joins. Characteristics:
- Scheduler/apiserver join carry trivial payloads (`NULL` attrs); worker join carries a small
`hostattrs_t`. All reuse the daemons' existing startup join path — no new heavy machinery.
- On rejoin, apiserver/scheduler receive **full** state (e.g. the full job list), not a delta; for a
large cluster with many workers this is sizable but infrequent. This is simpler and correct, and
still cheaper than a full-history replace with no snapshot (cf. IBM LSF).
- **Herd control**: the jittered backoff (6.3) staggers simultaneous reconnects after a failover.
**Deferred optimization (YAGNI):** a lightweight "re-attach" (session-token hello letting the primary
skip re-pushing already-replicated state, or delta-only state transfer) is intentionally **not**
designed now — delta reconciliation is non-trivial and full-state-on-rejoin is acceptable. Revisit
only if failover-rejoin latency is *measured* as a problem at scale. Recorded in `roadmap.md` backlog.
## Testing Strategy
### Unit Tests (`test_libxce_retry.c`, realigned)
- `xce_failover_classify`: REDIRECT+addr→ROTATE_ADDR; REDIRECT/STANDBY no-addr→ROTATE;
NOMASTER/NOT_READY→BACKOFF; AUTH/PERM/INVALID/NOENT→TERMINAL; 0→SUCCESS (2.5, 4.1-4.5).
- `xcecli_retry_next` loop: NOMASTER until budget → returns `-E_XCE_NOT_READY` (3.3); becomes primary
mid-loop → success, sticky cursor advanced (2.4, 3.2); non-retriable → immediate surface (4.5).
- Stale-reply avoidance (CI1): after a reconnect (old fd closed first), a stale entry left in `wbuf`
is drained before the next attempt, so no prior-attempt status is consumed; no sequence matching (1.2, 5.3).
- Context state (CI2): after a transport error drives `XCE_CTX_ERR`, a successful reconnect restores
`XCE_CTX_RUN` and the retry completes — never surfaces `-EINVAL` (1.2, 3.2).
- Idempotent write (CI3): `DUP_JOB` on a re-sent `jobsub` (attempt>0) → success; on attempt 0 →
surfaced as duplicate (5.2).
- EndpointRotator: redirect address used first (8.3); absent → next master (8.4); rotation reuses
same endpoint object.
- RetryConfig: NULL cfg → defaults (6.2); jitter staggers concurrent retries (6.3, reuse 50-client).
### Integration / E2E (3-node, `/xce-deploy-ha` + `xce_test`)
- Submit jobs, kill primary, `joblist`: returns cluster-not-ready during the election window, then
transparently succeeds against the new primary with no error (1.2, 3.2, 3.3) — the bug reproduction.
- Redirect path: contact a standby directly → client lands on primary in one hop via the populated
redirect address (8.1, 8.3).
- Write failover: `jobsub` across a failover → exactly one job created (5.1, 5.2 via controller idempotency).
### Controller Unit Test
- `handle_join` redirect path populates `redirect_host/port` from `ha.primary_host/port` (8.1);
unknown primary still yields `E_XCE_NOMASTER` (8.2).

179
.kiro/specs/libxce-failover/requirements.md Normal file
View file

@ -0,0 +1,179 @@
# Requirements Document
## Introduction
This feature makes XCE controller primary failover transparent to client applications built on
libxce (the XCE client library), including the worker, scheduler, and API server daemons and
the `xce_test` REPL.
Today, when the connected controller primary is killed in an HA cluster, libxce keeps using the
dead connection and returns an opaque invalid-argument error indefinitely: it never fails over
to the newly elected primary and never reports a clear "cluster not ready" during the election
window. The upstream requirement (`ha-config-bootstrap` Requirement 7) delivered only a
standalone retry primitive that no production request path invokes and that cannot discover a
new primary after the connected one dies. This spec completes that intent on the client side:
it routes all client requests through transparent retry/failover, rotates the configured master
list to locate the new primary, and surfaces a distinct cluster-not-ready error only when an
election outlasts the retry budget.
## Boundary Context
- **In scope**: client-side transparent failover for all libxce request types; discovery of the
new primary by rotating the configured master list, using the redirect address (when present) as
a best-effort hint to try first; bounded transparent retry that surfaces a distinct
cluster-not-ready error on budget exhaustion; failover for both read and write requests;
configurable retry/backoff parameters with safe defaults; operator-visible logging of failover
events; one small controller change so that redirect responses carry the primary's host and port
(the wire schema and reply struct already define these fields); end-to-end verification with the
`xce_test` client (the controller test harness, and the original reproduction case for this bug).
- **Out of scope**: controller-side election, log replication, commit barrier, and write
idempotency (consumed, not changed); introducing new probe or transport message types or new
reply fields (the redirect address fields already exist); per-request retry opt-in flags (writes
fail over by default); **adoption by the worker, scheduler, and apiserver daemons** — their
master-list configuration and liveness-safe handling of bounded blocking are owned by the
follow-on spec `component-failover-adoption` (depends on this spec).
- **Adjacent expectations**: libxce classifies controller responses by the distinct statuses the
controller actually emits — a *redirect* status (a primary exists but this node is a standby), a
*no-primary* status (this standby knows no primary yet), a *not-ready* status (the controller is
still initializing or electing), and a *standby* status (the operation is primary-only). With
the in-scope controller change, a redirect response carries the primary's host and port; libxce
treats this as a **best-effort hint** — it tries the hinted address first, and rotates its
configured master list when the hint is absent, stale, or unreachable. Correctness never depends
on the hint. libxce relies on controllers to deduplicate re-sent
writes through their commit-barrier idempotency guards, and on the configured master list (e.g.
`set masters host:port,host:port,host:port`) being the full set of controller candidates.
- **Stateful consumers**: a pure client (`xce_test`) needs only reconnect + resend. Stateful daemons
(apiserver, worker, scheduler) hold a session established via join; after a failover they must
re-establish it on the new primary (and a worker re-syncs its running jobs). libxce provides a
reconnect/rejoin hook for this; the hook *content* is owned by `component-failover-adoption` and
`worker-rejoin-resync`. Fire-and-forget notifications have no reply, so libxce cannot detect a
standby for them or guarantee delivery across a failover — redelivery is `worker-rejoin-resync`.
## Requirements
### Requirement 1: Transparent Failover for All Client Requests
**Objective:** As a developer of a libxce-based daemon, I want every controller request to fail
over automatically inside the library, so that I do not write ad-hoc retry or reconnect loops in
caller code.
#### Acceptance Criteria
1. The libxce client library shall route every controller request, both read/query and
mutating/write requests, through the transparent retry-and-failover path.
2. When controller failover occurs and a new primary becomes available, the libxce client
library shall complete the next request against the new primary without requiring any change
to caller code.
3. While failover handling is active, the libxce client library shall preserve the existing
public API call signatures and success/error return conventions so that existing callers
continue to compile and run unchanged.
### Requirement 2: New-Primary Discovery Across the Configured Master List
**Objective:** As a client application, I want libxce to find the new primary after the connected
one dies, so that my requests continue to reach an authoritative controller.
#### Acceptance Criteria
1. If the connected controller becomes unreachable through connection refused, connection
timeout, or peer reset, the libxce client library shall attempt the other controllers in its
configured master list to locate the current primary.
2. When a contacted controller responds with a redirect status indicating that a primary exists
elsewhere, the libxce client library shall locate the primary by rotating its configured master
list, trying the redirect address (when the response carries one) first as a best-effort hint.
3. When a contacted controller responds that the requested operation is primary-only (a standby
status), the libxce client library shall treat it as a redirect and attempt the other
controllers in its configured master list.
4. When a controller accepts and serves the request as primary, the libxce client library shall
treat it as the current primary and direct subsequent requests to it.
5. The libxce client library shall not consider a request successful when it was answered by a
controller reporting a redirect, standby, no-primary, or not-ready status.
### Requirement 3: Election-Window Behavior and Cluster-Not-Ready Reporting
**Objective:** As a client application, I want short failovers hidden and longer outages reported
clearly, so that I can distinguish "retry later" from a real failure.
#### Acceptance Criteria
1. While every reachable controller in the configured master list reports no primary exists (a
no-primary or not-ready status), the libxce client library shall retry across the configured
controllers with exponential backoff until the retry budget is exhausted.
2. When a primary becomes available before the retry budget is exhausted, the libxce client
library shall complete the request successfully and return no error to the caller.
3. If the retry budget is exhausted while no controller has become primary, the libxce client
library shall return a distinct cluster-not-ready error to the caller.
4. The cluster-not-ready error shall be distinguishable by the caller from transport connection
errors and from request-validation errors, so that a no-primary window is never reported as an
opaque invalid-argument error.
### Requirement 4: Error Classification — Retriable vs. Terminal
**Objective:** As a client application, I want transient controller errors retried and permanent
errors surfaced immediately, so that I neither hang on unrecoverable errors nor fail on recoverable ones.
#### Acceptance Criteria
1. If a controller returns a no-primary or not-ready status, the libxce client library shall retry
the request with exponential backoff.
2. If a controller returns a redirect status (a primary exists elsewhere), the libxce client
library shall locate and switch to the primary per Requirement 2 and retry promptly, without
waiting for the full backoff delay used for the no-primary case.
3. If a controller returns a standby status (the operation is primary-only), the libxce client
library shall switch to another configured controller per Requirement 2 and retry.
4. If a transport-level connection refused, timeout, or peer reset occurs, the libxce client
library shall reconnect or rotate to another configured controller and retry.
5. If a controller returns a non-retriable error, specifically authentication failure, permission
denied, invalid request, or not found, the libxce client library shall surface that error to
the caller immediately without retry.
### Requirement 5: Read and Write Failover Semantics
**Objective:** As a client application, I want both queries and mutating operations to survive
failover without duplicating side effects, so that failover is safe as well as transparent.
#### Acceptance Criteria
1. The libxce client library shall apply transparent failover to both read/query requests and
mutating/write requests, such as job submission.
2. When a write request is retried after a failover, the libxce client library shall not cause the
operation to be applied more than once, relying on the controller's idempotency handling for
re-sent writes.
3. While failing over, the libxce client library shall not silently drop a request; it shall
either complete the request against a primary or surface a distinct error to the caller.
### Requirement 6: Configurable Retry Parameters and Thundering-Herd Avoidance
**Objective:** As an operator, I want retry behavior tunable with safe defaults and protection
against synchronized retries, so that failover does not overwhelm a freshly elected primary.
#### Acceptance Criteria
1. The libxce client library shall provide configurable retry parameters: maximum retries,
initial delay, maximum delay, and backoff multiplier.
2. Where no retry configuration is supplied, the libxce client library shall apply safe
production defaults.
3. When many clients fail over simultaneously, the libxce client library shall stagger their
retry attempts through randomized backoff so that they do not retry in lockstep against the new
primary.
### Requirement 7: Failover Observability
**Objective:** As an operator, I want failover activity visible in logs, so that I can confirm
and diagnose client failover during a controller outage.
#### Acceptance Criteria
1. When the libxce client library fails over to a different controller, it shall log the failover
event including the target controller endpoint.
2. When the libxce client library returns a cluster-not-ready error after exhausting its retry
budget, it shall log the exhaustion including the number of attempts made.
### Requirement 8: Redirect Responses Carry the Primary Address (used as a hint)
**Objective:** As a client application, I want a standby's redirect to *hint* where the primary is,
so libxce can usually reach it in one hop while still working correctly without the hint.
#### Acceptance Criteria
1. When a standby controller redirects a client because it knows a primary exists elsewhere, the
controller shall include the primary's host and port in the redirect response.
2. While a standby controller does not know the primary's address, the controller shall return a
no-primary status rather than a redirect that omits the address.
3. When the libxce client library receives a redirect response that includes a primary address, it
shall try that address first as a best-effort hint.
4. If the redirect carries no address, or the hinted address is unreachable or not the primary, the
libxce client library shall fall back to rotating its configured master list; correctness shall
not depend on the hint.

127
.kiro/specs/libxce-failover/research.md Normal file
View file

@ -0,0 +1,127 @@
# Research Log — libxce-failover
## Discovery Scope
Light, integration-focused discovery (extension of existing libxce). Goal: find the exact
seams to wire transparent failover into the client request/reply path. Verified by reading the
endpoint, client-config, request-core, reply, and controller-redirect code paths.
## Key Findings
### F1 — The async request/reply model determines the chokepoint (DECISIVE)
`xce_request(ep, req)` (request.c:317) only **encodes + enqueues** a packet — fire-and-forget.
The controller's reply (and its `status`) is delivered asynchronously: the endpoint's
`msg_handler` runs `handle_reply()` (xcectx.c:30) which signals the caller's wait channel, and
public ops block on `cbuf_get(&ctx->wbuf, &status)` (xceapi.c:116). Therefore the controller
status (`E_XCE_REDIRECT`/`NOMASTER`/`NOT_READY`/`STANDBY`) is observable only at the
**send-and-wait round trip**, not at `xce_request`. Failover orchestration must wrap that round
trip, where the `api_ctx_t` is in scope.
### F2 — `xce_request_with_retry()` production branch is a no-op stub
request.c:1972-1981: with no test hook installed, it does `xce_request(ep,req); return 0;`. The
entire retry/classify/backoff loop executes only under `xce_retry_send_hook` (tests). So the
"done" primitive cannot fail over against a real controller. Its hook contract
(`int hook(ep, req, reply_out)`) fakes a synchronous reply that production never produces.
### F3 — Redirect address is plumbed in the wire/struct but never populated or surfaced
- Wire schema has the fields: `controller.capnp:568-569` (`redirectHost`/`redirectPort`).
- Reply struct has them: `typemap.h:225-226` (`char *redirect_host; uint16_t redirect_port;`).
- Controller never sets them: every redirect site (`join.c:264`) sends `E_XCE_REDIRECT` with
`data_which = nodata`; grep finds **zero** production assignments to `reply.redirect_*`.
- Client never surfaces them: `handle_reply()` (xcectx.c:34) returns only `reply.status`; the
redirect fields are dropped before any caller sees them.
### F4 — Context already holds everything failover needs
`api_ctx_t` (client.c:23): `{ xce_client_config_t *conf; endpoint_t ep; xce_ctx_t ctx; }`.
`xce_client_config_t` (client.h:16): `masters[N_MAX_MASTERS]` + `master_ports[]` + `nmasters`.
So the master list, the endpoint to reconnect, and the reply channel are all reachable from one
context object. Missing: a current-master cursor, the retry config, and captured redirect target.
### F5 — Endpoint reconnect is straightforward and reusable
`endpoint_connect(ep, host, port)` (netio.c:659) does socket connect + SSL handshake on the
**same** `endpoint_t`. Reconnecting to a different master reuses the same endpoint object, so the
`msg_handler`/`owner` wiring that routes replies to `ctx->wbuf` stays intact.
### F6 — `xcecli_start()` only ever connects `masters[0]` (client.c:122)
No primary discovery at startup; single endpoint, never re-selected. Confirms the gap.
### F7 — Original EINVAL source
`xce_get_with_condition()` returns `-EINVAL` when `ctx->state != XCE_CTX_RUN` (xceapi.c:104).
After the primary dies the connection breaks and the ctx is left non-RUN, so every subsequent
call short-circuits to `-EINVAL` — the opaque error in the bug report. Failover must repair the
endpoint/ctx state instead of leaving it dead.
## Decisions (synthesis)
- **D1 (final — client-layer, supersedes the earlier xceapi round-trip idea):** failover lives in
the **`client.c` `xcecli_*` layer**, not at `xce_request` and not at `xceapi.c`. Evidence: every
consumer (worker `controller_connector.c`, scheduler `scheduler.c`, apiserver `clib.go`, xce_test)
calls `xcecli_*`, never `xce_*` directly; and only `api_ctx_t` (client.c) holds the master list —
`xce_ctx_t` (xceapi) does not. So `client.c` is both the universal surface and the only layer with
conf+ep+ctx co-located. **`xce_*` stay one-shot/fail-fast and unchanged**; each `xcecli_*` wraps
its `xce_*` call in a `do/while` driven by a shared `xcecli_retry_next` helper
(`src/lib/client_failover.c` + `client_internal.h`). Trade-off accepted (user): ~33 wrappers gain
a small loop (more LOC) but the policy is one helper and no async/thunk/plumbing changes — simpler
and lower-risk. The orphaned `xce_request_with_retry()` is removed, not wired.
- **D2:** retire the no-op production branch of `xce_request_with_retry()`. Keep `xce_retry_config_t`
+ `XCE_RETRY_DEFAULTS`. Realign the unit tests (and hook contract) to the real status taxonomy
and the round-trip shape (allowed by Req 8 / requirements decision; tracked in bug-608).
- **D3:** `handle_reply()` captures `reply.redirect_host/port` + status into the ctx (deep-copy the
host string before the decoded reply is freed) so the failover loop can prefer the address.
- **D4:** on endpoint error/RDHUP, the err path must release the pending `cbuf_get` with a
transport error so the loop reconnects instead of hanging.
- **D5:** controller populates `reply.redirect_host/port` from `ha.primary_host/port` at every
`E_XCE_REDIRECT` site (join.c confirmed sole site; audit during implementation).
- **D6:** budget-exhaustion mapping — no-primary cause → return distinct `-E_XCE_NOT_READY`
("cluster not ready"); transport-only cause → return last transport error. Never `-EINVAL`.
### Decisions added after design review (validate-design pass)
- **D7 (CI1 — reply correlation):** `ctx->wbuf` is single-slot, status-only
(`cbuf_init(...,1,sizeof(int))`, xcectx.c:185). `handle_reply` records `reply.seq` into the ctx;
`xcecli_retry_next` drains any stale `wbuf` entry after reconnect (and reconnect closes the old fd
first → no further old replies), so the next `xce_*` attempt cannot consume a prior-attempt reply.
- **D8 (CI2 — context state reset):** the existing transport-error path sets `ctx->state =
XCE_CTX_ERR` (xcectx.c:148), which the per-op guard rejects as `-EINVAL` (xceapi.c:14,…) — the
reported bug. `xcecli_retry_next` restores `XCE_CTX_RUN` after a successful reconnect, before the
next `xce_*` attempt, so the unchanged per-op guard passes. `xceapi.c` is not modified.
### Simplification pass (goal: keep libxce easy to use)
- **D13 (redirect = hint, not contract):** the controller still populates the redirect address
(forward-compatible, kept), but the client treats it as a **best-effort hint** — try it first if
present, else rotate the master list; correctness never depends on it. This removes the heavy
machinery the address-as-contract demanded: **drop the `reply_seq` correlation** (CI1 becomes
drain-after-close only — reconnect closes the old fd first, then drain `wbuf`), and **collapse the
classifier from 5 actions to 4** (`SUCCESS` / `RETRY_NOW` / `RETRY_BACKOFF` / `TERMINAL`) — the
hint is consumed by `EndpointRotator`, not a separate classifier branch. Net: fewer fields, no
lifetime-hazard dependency, simpler classification, same user-visible behavior. The rejoin hook
and per-ctx lock are kept (real needs). Rationale: with 3 masters and rare failover, rotation is
cheap; the hint is a latency nicety, not a correctness mechanism.
### Decisions added after 2nd design review (validate-design pass 2)
- **D10 (CI1 — op categories):** the ~33 `xcecli_*` ops split into reply-read (full failover,
`is_write=false`), reply-write (full failover, `is_write=true`, DUP-replay→success), and
fire-and-forget (`notify_*`/`heartbeat`/`wakeup` — `xce_*_req` only, no `cbuf_get`/status). The
failover classifier applies only to reply-bearing ops; fire-and-forget gets transport-reconnect
best-effort, and redelivery of lost notifies is deferred to `worker-rejoin-resync`.
- **D11 (CI2 — concurrency):** apiserver shares one ctx across goroutines (`clib.go:182`); add a
`failover.lock` taken per `xcecli_*` op so reconnect/rotate cannot race another in-flight call.
- **D12 (rejoin hook — user requirement):** stateful daemons (apiserver/worker/scheduler) need a
rejoin/state-exchange on reconnect; a worker also re-syncs its running-jobs state. libxce adds
`xcecli_set_reconnect_handler()`; the failover loop invokes it after reconnect, before resume. The
hook *content* is owned by `component-failover-adoption` (join) and `worker-rejoin-resync` (job
state). Pure clients (`xce_test`) register none.
- **D9 (CI3 — idempotent-write replay):** job submit carries a **client-generated** `jobid`
(`job.c:56`) with a server `-EEXIST`/`E_XCE_DUP_JOB` replay guard (`job.c:95`). The classifier maps
`DUP_JOB` on a re-sent write (`is_write && attempt>0`) to success; on attempt 0 it is a genuine
duplicate (terminal). This is what makes 5.2 actually satisfiable rather than assumed.
## Risks
- **R-async-blocking:** the round trip blocks longer under retry. Acceptable for CLI/apiserver
(already blocking); for async C daemons it's the `component-failover-adoption` spec's concern
(out of boundary here).
- **R-reply-routing on reconnect:** re-sending after reconnect must correlate the reply to the
same ctx wait channel; mitigated by reusing the same `endpoint_t` (F5) and matching on `seq`.
- **R-redirect-lifetime:** `redirect_host` is a decoded capnp string freed with the reply; must be
copied into a ctx-owned buffer (D3) to avoid use-after-free (cf. the `init_reply_by_req`
memset hazard already documented at request.c:243-250).

22
.kiro/specs/libxce-failover/spec.json Normal file
View file

@ -0,0 +1,22 @@
{
"feature_name": "libxce-failover",
"created_at": "2026-06-16T10:30:55Z",
"updated_at": "2026-06-16T10:52:00Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": false
}

103
.kiro/specs/libxce-failover/tasks.md Normal file
View file

@ -0,0 +1,103 @@
# Implementation Plan
- [ ] 1. Foundation: failover state, captured-reply fields, and cleanup
- [x] 1.1 Establish shared failover state and reply-capture fields
- Expose the client context to a new client-internal header so the failover helper can operate on it
- Add per-context failover state: a lock, the current-master cursor, retry parameters (with safe defaults), and a reconnect/rejoin callback slot
- Add fixed-size redirect-hint host/port fields to the client reply context (advisory hint; no sequence correlation)
- Initialize all new state when the client starts; keep the retry-config defaults; remove the orphaned, never-wired retry primitive
- Done: libxce builds with the new per-context state present and the dead primitive removed; retry defaults are reachable
- _Requirements: 6.1, 6.2_
- [ ] 1.2 Realign the retry test harness to the new model (RED)
- Retarget the existing libxce retry tests from the removed primitive to the new classifier and client-wrapper retry path
- Add fixtures simulating each controller status (redirect with and without address, no-primary, not-ready, standby, duplicate-job) and transport failures, plus a fake multi-master list and a shared-context concurrency harness
- Done: the test file compiles against the new symbols and fails (RED) because the implementation does not yet exist
- _Requirements: 2.5, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 8.3, 8.4_
- [ ] 2. Core: status classification, endpoint rotation, reply-path capture
- [ ] 2.1 Implement status classification
- Map each controller status and transport result to one of four actions: success, retry-now (redirect/standby/transport — rotate without backoff), retry-after-backoff (no-primary/not-ready), terminal
- Never classify redirect, standby, no-primary, or not-ready as success
- Treat a duplicate-job/exists status on a re-sent write as success (idempotent replay); on a first attempt treat it as a genuine duplicate surfaced to the caller
- Done: classification unit tests pass for every status and for both duplicate-replay cases
- _Requirements: 2.5, 4.1, 4.2, 4.3, 4.5, 5.2_
- _Boundary: StatusClassifier_
- [ ] 2.2 Implement endpoint reconnect and master rotation
- On rotate, try the captured redirect-hint address first when one is set (then clear it); otherwise advance the master cursor to the next configured master and reconnect
- Close the stale connection first and reuse the same endpoint object so reply routing stays intact; a failed hint simply falls through to the next master
- Done: rotation unit tests show hint-first then next-master fallback, reusing a single endpoint object, with correctness independent of the hint
- _Requirements: 2.1, 2.2, 2.3, 4.4, 8.4_
- _Boundary: EndpointRotator_
- _Depends: 1.1_
- [ ] 2.3 (P) Capture redirect on the reply path and wake the wait on endpoint error
- In the reply handler, copy the redirect-hint host/port into the client reply context when a redirect reply carries one, before the reply is freed; leave empty otherwise (advisory hint, no sequence correlation)
- On endpoint error or peer reset, release any pending reply wait with a transport error so the in-flight call returns instead of hanging
- Keep the status flowing through the existing wait-channel path so the public API layer stays unchanged
- Done: after a redirect reply the captured hint address is readable from the context; a simulated endpoint error releases a pending wait with a transport error (no hang); the API layer is untouched
- _Requirements: 2.2, 4.4, 8.3_
- _Boundary: RedirectCapture, ReplyWaitWake_
- _Depends: 1.1_
- [ ] 3. Core: the retry/failover decision helper
- [ ] 3.1 Implement the decision loop (classify, rotate, backoff, budget)
- Combine classification, rotation, and the captured redirect into one decision: success and terminal stop the loop; redirect, standby, and transport results rotate without backoff; no-primary and not-ready apply jittered backoff
- Track the retry budget; on exhaustion with no primary, return the distinct cluster-not-ready code
- Update the sticky current-master on a successful served-as-primary reply; log the target on each rotation and the attempt count on budget exhaustion
- Done: helper-level tests show no-primary-until-budget returns cluster-not-ready, become-primary returns success with the cursor advanced, and redirect/transport rotate without the no-primary backoff
- _Requirements: 1.2, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.4, 6.3, 7.1, 7.2_
- _Depends: 2.1, 2.2, 2.3_
- _Boundary: xcecli_retry_next_
- [ ] 3.2 Implement reconnect recovery: context-state reset and stale-reply drain
- After a successful reconnect, restore the context from its error state to its running state before the next attempt, so a transport error never leaves the context returning an invalid-argument error
- Drain any stale reply left in the wait channel after reconnect so the next attempt cannot consume a prior-attempt reply
- Done: a transport error followed by reconnect no longer surfaces an invalid-argument error and the context is reusable; a unit test injecting a stale reply confirms it is drained, not returned
- _Requirements: 1.2, 4.4_
- _Depends: 3.1_
- _Boundary: xcecli_retry_next_
- [ ] 4. Core: reconnect/rejoin hook
- [ ] 4.1 Add the reconnect/rejoin hook mechanism
- Provide a client API to register a reconnect callback and argument, stored in the failover state
- Invoke the callback after each successful reconnect to a new primary and before resuming the interrupted op; treat a failing callback as a failed attempt that rotates or backs off per its result
- Done: with a registered hook a simulated failover invokes it before resume; with no hook the loop resumes directly
- _Requirements: 1.2_
- _Depends: 3.1, 3.2_
- [ ] 5. Integration: wrap the client ops; controller redirect address
- [ ] 5.1 Wrap reply-bearing client ops in the failover loop under the per-context lock
- Wrap each reply-bearing read and write op so it runs its single-shot call inside the decision loop, holding the per-context lock for the loop's duration
- Tag write ops so the idempotent-replay rule applies; preserve every public signature and success/error convention
- Done: existing callers compile unchanged; a reply-bearing op transparently completes against a new primary after a simulated failover; concurrent use of one context is serialized
- _Requirements: 1.1, 1.3, 2.4, 5.1, 5.3_
- _Depends: 3.1, 3.2, 4.1_
- _Boundary: XcecliRetryLoop, FailoverState_
- [ ] 5.2 Apply best-effort reconnect to fire-and-forget ops
- Give the notify, heartbeat, and wakeup ops a transport-reconnect-only variant: reconnect on a known-dead connection, with no status classification and no no-primary backoff
- Done: a fire-and-forget op on a dead connection triggers a reconnect attempt and never blocks on the no-primary backoff budget; lost-notification redelivery remains out of scope
- _Requirements: 1.1, 5.3_
- _Depends: 3.1_
- _Boundary: XcecliRetryLoop_
- [ ] 5.3 (P) Populate the primary address in controller redirect replies
- On the standby redirect path, fill the reply's redirect host and port from the known primary address; keep returning the no-primary status when the primary is unknown
- Audit for any other redirect-emitting sites and apply the same population
- Add a controller unit test: the redirect path populates the address, and an unknown primary still yields the no-primary status
- Done: the controller test passes and a standby redirect reply carries the primary address
- _Requirements: 8.1, 8.2_
- _Boundary: ControllerRedirectAddress_
- [ ] 6. Validation: unit suite and live failover
- [ ] 6.1 Bring the realigned unit suite green
- Make the retry tests pass for classification, budget-to-cluster-not-ready, sticky cursor, hint-first then rotate, duplicate-replay-to-success, jitter and many-client staggering, stale-reply drain, context-state recovery, and concurrent shared-context use
- Done: the full libxce retry unit suite passes under the project's test and ASAN build
- _Requirements: 2.4, 2.5, 3.2, 3.3, 4.1, 4.2, 4.3, 4.4, 4.5, 5.2, 6.2, 6.3, 8.3, 8.4_
- _Depends: 5.1, 5.2_
- [ ] 6.2 Live 3-node failover end-to-end
- On a 3-node cluster, submit jobs, kill the primary, and verify joblist reports cluster-not-ready during the election window then transparently succeeds against the new primary
- Verify a request sent to a standby lands on the primary — in one hop when the redirect hint is present, otherwise via master-list rotation
- Verify a job submission spanning a failover creates exactly one job
- Done: the original reproduction passes — no invalid-argument error, transparent recovery, and a single job created
- _Requirements: 1.2, 3.2, 3.3, 5.1, 5.2, 8.1, 8.3_
- _Depends: 5.1, 5.3_
## Implementation Notes
- 1.1: design said `XCE_MAXHOST_LEN`; real constant is `XCE_MAXHOSTNAME_LEN` (config.h, =256) — design corrected. `api_ctx_t` now in private `client_internal.h`; `xcecli_reconnect_cb` typedef lives there until task 4.1 adds the public setter. Build: `cd src && nix develop --command bash tools/xce-build.sh lib`.

26
.wolf/anatomy.md
View file

@ -1,23 +1,25 @@
# anatomy.md # anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-06-16T07:17:57.828Z > Auto-maintained by OpenWolf. Last scanned: 2026-06-16T08:19:04.353Z
> Files: 507 tracked | Anatomy hits: 0 | Misses: 0 > Files: 516 tracked | Anatomy hits: 0 | Misses: 0
## ./ ## ./
- `.clang-format` (~226 tok) - `.clang-format` (~226 tok)
- `.gitignore` — Git ignore rules (~88 tok) - `.gitignore` — Git ignore rules (~88 tok)
- `.mcp.json` (~7 tok)
- `.opencode.json` (~55 tok) - `.opencode.json` (~55 tok)
- `CLAUDE.md` — OpenWolf (~1400 tok) - `CLAUDE.md` — OpenWolf (~1400 tok)
- `package-lock.json` — npm lock file (~1039 tok) - `package-lock.json` — npm lock file (~1039 tok)
- `package.json` — Node.js package manifest (~1 tok) - `package.json` — Node.js package manifest (~1 tok)
- `README.md` — Project documentation (~612 tok) - `README.md` — Project documentation (~612 tok)
- `uninstall-code-review-graph.sh` — Fully uninstall `code-review-graph` (the Python/uv MCP tool). (~627 tok)
## .claude/ ## .claude/
- `settings.json` (~497 tok) - `settings.json` (~497 tok)
- `settings.json.bak` (~329 tok) - `settings.json.bak` (~329 tok)
- `settings.local.json` (~187 tok) - `settings.local.json` (~186 tok)
- `ucai-formatter-cache.local.json` (~17 tok) - `ucai-formatter-cache.local.json` (~17 tok)
## .claude/rules/ ## .claude/rules/
@ -296,6 +298,11 @@
- `spec.json` (~142 tok) - `spec.json` (~142 tok)
- `tasks.md` — Implementation Plan: ha-convergence-recovery (~5679 tok) - `tasks.md` — Implementation Plan: ha-convergence-recovery (~5679 tok)
## .kiro/specs/ha-elect-ctx-lifecycle/
- `brief.md` — Discovery Brief: ha-elect-ctx-lifecycle (~774 tok)
- `spec.json` (~130 tok)
## .kiro/specs/ha-heartbeat-liveness/ ## .kiro/specs/ha-heartbeat-liveness/
- `design.md` — Design Document: ha-heartbeat-liveness (~5409 tok) - `design.md` — Design Document: ha-heartbeat-liveness (~5409 tok)
@ -319,7 +326,7 @@
- `requirements.md` — Requirements Document (~3604 tok) - `requirements.md` — Requirements Document (~3604 tok)
- `research.md` — Gap Analysis: ha-prevote (~6077 tok) - `research.md` — Gap Analysis: ha-prevote (~6077 tok)
- `spec.json` (~126 tok) - `spec.json` (~126 tok)
- `tasks.md` — Implementation Plan (~3129 tok) - `tasks.md` — Implementation Plan (~3197 tok)
## .kiro/specs/ha-repl-hardening/ ## .kiro/specs/ha-repl-hardening/
@ -710,6 +717,17 @@
- `3c576da7137df22cc5fbd982fd231d9d40d374a77c7d618081ba7018f5eec36c.json` (~4654 tok) - `3c576da7137df22cc5fbd982fd231d9d40d374a77c7d618081ba7018f5eec36c.json` (~4654 tok)
- `3ce0572164f9daaf2eee6a108f57bb947fb3d812c5578c7451eed78062e38b68.json` (~8024 tok) - `3ce0572164f9daaf2eee6a108f57bb947fb3d812c5578c7451eed78062e38b68.json` (~8024 tok)
## src/controller/
- `cluster.c` — include "config.h" (~15511 tok)
- `ha.c` — define _GNU_SOURCE (~52565 tok)
- `ha.h` — ifndef HA_H (~14911 tok)
- `persist.c` — define _GNU_SOURCE (~18295 tok)
## src/include/
- `persist.h` — pragma once (~2668 tok)
## src/lib/testsuite/ ## src/lib/testsuite/
- `test_raft.c` — include <errno.h> (~36211 tok) - `test_raft.c` — include <errno.h> (~36211 tok)

2
.wolf/cron-state.json
View file

@ -1,5 +1,5 @@
{ {
"last_heartbeat": "2026-06-16T07:25:23.733Z", "last_heartbeat": "2026-06-16T08:25:23.734Z",
"engine_status": "running", "engine_status": "running",
"execution_log": [ "execution_log": [
{ {

176
.wolf/hooks/_session.json
View file

@ -1,143 +1,131 @@
{ {
"session_id": "session-2026-06-16-2344", "session_id": "session-2026-06-16-0057",
"started": "2026-06-16T06:44:27.944Z", "started": "2026-06-16T07:57:38.410Z",
"files_read": { "files_read": {
"/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/requirements.md": { "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json": {
"count": 6, "count": 2,
"tokens": 3604, "tokens": 7,
"first_read": "2026-06-16T06:45:04.651Z" "first_read": "2026-06-16T07:58:57.528Z"
}, },
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.c": { "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/commit.c": {
"count": 9,
"tokens": 0,
"first_read": "2026-06-16T06:45:17.576Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_connection_reuse.c": {
"count": 1, "count": 1,
"tokens": 0, "tokens": 0,
"first_read": "2026-06-16T06:45:26.432Z" "first_read": "2026-06-16T07:58:58.420Z"
}, },
"/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md": { "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/worker.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-06-16T07:59:27.185Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/join.c": {
"count": 1,
"tokens": 0,
"first_read": "2026-06-16T07:59:27.920Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c": {
"count": 4, "count": 4,
"tokens": 3129, "tokens": 0,
"first_read": "2026-06-16T06:48:13.339Z" "first_read": "2026-06-16T07:59:43.058Z"
}, },
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_prevote.c": { "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/meson.build": {
"count": 2,
"tokens": 7618,
"first_read": "2026-06-16T06:48:14.326Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_failover_prevote.c": {
"count": 2,
"tokens": 3896,
"first_read": "2026-06-16T06:48:24.387Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_vote_gate.c": {
"count": 1, "count": 1,
"tokens": 9075, "tokens": 0,
"first_read": "2026-06-16T06:48:24.556Z" "first_read": "2026-06-16T08:00:30.531Z"
}, },
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/controller_test.c": { "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/controller_test.c": {
"count": 1, "count": 1,
"tokens": 1469,
"first_read": "2026-06-16T06:48:31.170Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/design.md": {
"count": 4,
"tokens": 6905,
"first_read": "2026-06-16T06:48:40.155Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/raft/raft.c": {
"count": 3,
"tokens": 0, "tokens": 0,
"first_read": "2026-06-16T06:52:15.949Z" "first_read": "2026-06-16T08:00:40.931Z"
}, },
"/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/research.md": { "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json": {
"count": 1,
"tokens": 5295,
"first_read": "2026-06-16T07:10:16.063Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c": {
"count": 2, "count": 2,
"tokens": 0, "tokens": 497,
"first_read": "2026-06-16T07:10:25.883Z" "first_read": "2026-06-16T08:06:45.661Z"
}, },
"/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/raft/raft.h": { "/home/rshen/projects/xce/worktrees/persistent_ha/src/include/persist.h": {
"count": 1, "count": 1,
"tokens": 0, "tokens": 0,
"first_read": "2026-06-16T07:10:26.237Z" "first_read": "2026-06-16T08:17:36.150Z"
},
"/tmp/claude-1000/-home-rshen-projects-xce-worktrees-persistent-ha/bffce5f2-ffc9-4dc1-b956-0d3d7223fc0d/tasks/bei9rto5x.output": {
"count": 1,
"tokens": 0,
"first_read": "2026-06-16T07:20:19.102Z"
} }
}, },
"files_written": [ "files_written": [
{ {
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md", "file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"action": "edit", "action": "create",
"tokens": 20, "tokens": 7,
"at": "2026-06-16T06:46:55.771Z" "at": "2026-06-16T07:59:01.938Z"
}, },
{ {
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_prevote.c", "file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"action": "edit", "action": "edit",
"tokens": 3353, "tokens": 11,
"at": "2026-06-16T06:50:23.587Z" "at": "2026-06-16T08:07:10.456Z"
}, },
{ {
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md", "file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"action": "edit", "action": "edit",
"tokens": 14, "tokens": 10,
"at": "2026-06-16T06:54:20.104Z" "at": "2026-06-16T08:07:18.236Z"
}, },
{ {
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_prevote.c", "file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"action": "edit", "action": "create",
"tokens": 3780, "tokens": 7,
"at": "2026-06-16T06:58:28.913Z" "at": "2026-06-16T08:09:05.843Z"
}, },
{ {
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md", "file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"action": "edit", "action": "create",
"tokens": 16, "tokens": 497,
"at": "2026-06-16T07:08:51.022Z" "at": "2026-06-16T08:09:28.949Z"
}, },
{ {
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c", "file": "/home/rshen/projects/xce/worktrees/persistent_ha/uninstall-code-review-graph.sh",
"action": "edit", "action": "create",
"tokens": 4104, "tokens": 672,
"at": "2026-06-16T07:12:02.527Z" "at": "2026-06-16T08:09:44.867Z"
}, },
{ {
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c", "file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/include/persist.h",
"action": "edit", "action": "edit",
"tokens": 142, "tokens": 74,
"at": "2026-06-16T07:12:14.641Z" "at": "2026-06-16T08:18:22.177Z"
}, },
{ {
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/research.md", "file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"action": "edit", "action": "edit",
"tokens": 1012, "tokens": 56,
"at": "2026-06-16T07:13:49.151Z" "at": "2026-06-16T08:18:33.020Z"
}, },
{ {
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md", "file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"action": "edit", "action": "edit",
"tokens": 17, "tokens": 411,
"at": "2026-06-16T07:17:57.840Z" "at": "2026-06-16T08:18:48.563Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"action": "edit",
"tokens": 166,
"at": "2026-06-16T08:18:57.385Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"action": "edit",
"tokens": 63,
"at": "2026-06-16T08:19:04.362Z"
} }
], ],
"edit_counts": { "edit_counts": {
".kiro/specs/ha-prevote/tasks.md": 4, ".mcp.json": 2,
"src/controller/testsuite/test_ha_prevote.c": 2, ".claude/settings.json": 3,
"src/lib/testsuite/test_raft.c": 2, "uninstall-code-review-graph.sh": 1,
".kiro/specs/ha-prevote/research.md": 1 "src/include/persist.h": 1,
"src/controller/persist.c": 4
}, },
"anatomy_hits": 10, "anatomy_hits": 1,
"anatomy_misses": 4, "anatomy_misses": 8,
"repeated_reads_warned": 24, "repeated_reads_warned": 5,
"cerebrum_warnings": 0, "cerebrum_warnings": 0,
"stop_count": 1 "stop_count": 3
} }

1
.wolf/memory.md
View file

@ -1798,3 +1798,4 @@ Implemented sub-spec 3/4 of ha-consistency-invariants autonomously (per-task imp
| 01:19 | Edited src/controller/persist.c | 4→6 lines | ~59 | | 01:19 | Edited src/controller/persist.c | 4→6 lines | ~59 |
| 15:55 | Fixed controller test timeout: interruptible fsync_worker sleep (condvar) so persist_stop joins promptly | persist.c, persist.h | suite 300s+ TIMEOUT → 10s, 0 failures | ~25k | | 15:55 | Fixed controller test timeout: interruptible fsync_worker sleep (condvar) so persist_stop joins promptly | persist.c, persist.h | suite 300s+ TIMEOUT → 10s, 0 failures | ~25k |
| 01:23 | Session end: 11 writes across 5 files (.mcp.json, settings.json, uninstall-code-review-graph.sh, persist.h, persist.c) | 9 reads | ~2478 tok | | 01:23 | Session end: 11 writes across 5 files (.mcp.json, settings.json, uninstall-code-review-graph.sh, persist.h, persist.c) | 9 reads | ~2478 tok |
| 01:24 | Session end: 11 writes across 5 files (.mcp.json, settings.json, uninstall-code-review-graph.sh, persist.h, persist.c) | 9 reads | ~2478 tok |

783
.wolf/token-ledger.json
View file

@ -2,14 +2,14 @@
"version": 1, "version": 1,
"created_at": "2026-05-21T04:35:43.577Z", "created_at": "2026-05-21T04:35:43.577Z",
"lifetime": { "lifetime": {
"total_tokens_estimated": 42216062, "total_tokens_estimated": 42334085,
"total_reads": 10106, "total_reads": 10168,
"total_writes": 15352, "total_writes": 15414,
"total_sessions": 113, "total_sessions": 114,
"anatomy_hits": 6986, "anatomy_hits": 7012,
"anatomy_misses": 2948, "anatomy_misses": 2984,
"repeated_reads_blocked": 31561, "repeated_reads_blocked": 31627,
"estimated_savings_vs_bare_cli": 384256039 "estimated_savings_vs_bare_cli": 384382023
}, },
"sessions": [ "sessions": [
{ {
@ -143860,6 +143860,773 @@
"repeated_reads_blocked": 24, "repeated_reads_blocked": 24,
"anatomy_lookups": 10 "anatomy_lookups": 10
} }
},
{
"id": "session-2026-06-16-2344",
"started": "2026-06-16T06:44:27.944Z",
"ended": "2026-06-16T07:49:34.801Z",
"reads": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/requirements.md",
"tokens_estimated": 3604,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.c",
"tokens_estimated": 0,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_connection_reuse.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 3129,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_prevote.c",
"tokens_estimated": 7618,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_failover_prevote.c",
"tokens_estimated": 3896,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_vote_gate.c",
"tokens_estimated": 9075,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/controller_test.c",
"tokens_estimated": 1469,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/design.md",
"tokens_estimated": 6905,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/raft/raft.c",
"tokens_estimated": 0,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/research.md",
"tokens_estimated": 5295,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c",
"tokens_estimated": 0,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/raft/raft.h",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/tmp/claude-1000/-home-rshen-projects-xce-worktrees-persistent-ha/bffce5f2-ffc9-4dc1-b956-0d3d7223fc0d/tasks/bei9rto5x.output",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/cluster.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.h",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
}
],
"writes": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 20,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_prevote.c",
"tokens_estimated": 3353,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 14,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_prevote.c",
"tokens_estimated": 3780,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 16,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c",
"tokens_estimated": 4104,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c",
"tokens_estimated": 142,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/research.md",
"tokens_estimated": 1012,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 17,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.c",
"tokens_estimated": 32,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.c",
"tokens_estimated": 13,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.h",
"tokens_estimated": 86,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/cluster.c",
"tokens_estimated": 449,
"action": "edit"
}
],
"totals": {
"input_tokens_estimated": 40991,
"output_tokens_estimated": 13038,
"reads_count": 16,
"writes_count": 13,
"repeated_reads_blocked": 26,
"anatomy_lookups": 10
}
},
{
"id": "session-2026-06-16-2344",
"started": "2026-06-16T06:44:27.944Z",
"ended": "2026-06-16T07:56:37.342Z",
"reads": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/requirements.md",
"tokens_estimated": 3604,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.c",
"tokens_estimated": 0,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_connection_reuse.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 3129,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_prevote.c",
"tokens_estimated": 7618,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_failover_prevote.c",
"tokens_estimated": 3896,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_vote_gate.c",
"tokens_estimated": 9075,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/controller_test.c",
"tokens_estimated": 1469,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/design.md",
"tokens_estimated": 6905,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/raft/raft.c",
"tokens_estimated": 0,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/research.md",
"tokens_estimated": 5295,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c",
"tokens_estimated": 0,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/raft/raft.h",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/tmp/claude-1000/-home-rshen-projects-xce-worktrees-persistent-ha/bffce5f2-ffc9-4dc1-b956-0d3d7223fc0d/tasks/bei9rto5x.output",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/cluster.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.h",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 497,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.local.json",
"tokens_estimated": 187,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/CLAUDE.md",
"tokens_estimated": 1400,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/skills/explore-codebase/skill.md",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
}
],
"writes": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 20,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_prevote.c",
"tokens_estimated": 3353,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 14,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_prevote.c",
"tokens_estimated": 3780,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 16,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c",
"tokens_estimated": 4104,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c",
"tokens_estimated": 142,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/research.md",
"tokens_estimated": 1012,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 17,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.c",
"tokens_estimated": 32,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.c",
"tokens_estimated": 13,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.h",
"tokens_estimated": 86,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/cluster.c",
"tokens_estimated": 449,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 11,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-elect-ctx-lifecycle/spec.json",
"tokens_estimated": 130,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 10,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.local.json",
"tokens_estimated": 20,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-elect-ctx-lifecycle/brief.md",
"tokens_estimated": 884,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 64,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.kiro/specs/ha-prevote/tasks.md",
"tokens_estimated": 73,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/CLAUDE.md",
"tokens_estimated": 25,
"action": "edit"
}
],
"totals": {
"input_tokens_estimated": 43075,
"output_tokens_estimated": 14255,
"reads_count": 20,
"writes_count": 21,
"repeated_reads_blocked": 26,
"anatomy_lookups": 13
}
},
{
"id": "session-2026-06-16-0057",
"started": "2026-06-16T07:57:38.410Z",
"ended": "2026-06-16T08:10:11.254Z",
"reads": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"tokens_estimated": 7,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/commit.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/worker.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/join.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 0,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/meson.build",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/controller_test.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 497,
"was_repeated": true,
"anatomy_had_description": false
}
],
"writes": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"tokens_estimated": 7,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 11,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 10,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"tokens_estimated": 7,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 497,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/uninstall-code-review-graph.sh",
"tokens_estimated": 672,
"action": "create"
}
],
"totals": {
"input_tokens_estimated": 504,
"output_tokens_estimated": 1204,
"reads_count": 8,
"writes_count": 6,
"repeated_reads_blocked": 4,
"anatomy_lookups": 1
}
},
{
"id": "session-2026-06-16-0057",
"started": "2026-06-16T07:57:38.410Z",
"ended": "2026-06-16T08:23:24.362Z",
"reads": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"tokens_estimated": 7,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/commit.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/worker.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/join.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 0,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/meson.build",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/controller_test.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 497,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/include/persist.h",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
}
],
"writes": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"tokens_estimated": 7,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 11,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 10,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"tokens_estimated": 7,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 497,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/uninstall-code-review-graph.sh",
"tokens_estimated": 672,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/include/persist.h",
"tokens_estimated": 74,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 56,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 411,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 166,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 63,
"action": "edit"
}
],
"totals": {
"input_tokens_estimated": 504,
"output_tokens_estimated": 1974,
"reads_count": 9,
"writes_count": 11,
"repeated_reads_blocked": 5,
"anatomy_lookups": 1
}
},
{
"id": "session-2026-06-16-0057",
"started": "2026-06-16T07:57:38.410Z",
"ended": "2026-06-16T08:24:49.631Z",
"reads": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"tokens_estimated": 7,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/commit.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/worker.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/join.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 0,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/meson.build",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/controller_test.c",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 497,
"was_repeated": true,
"anatomy_had_description": false
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/include/persist.h",
"tokens_estimated": 0,
"was_repeated": false,
"anatomy_had_description": false
}
],
"writes": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"tokens_estimated": 7,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 11,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 10,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.mcp.json",
"tokens_estimated": 7,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/.claude/settings.json",
"tokens_estimated": 497,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/uninstall-code-review-graph.sh",
"tokens_estimated": 672,
"action": "create"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/include/persist.h",
"tokens_estimated": 74,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 56,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 411,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 166,
"action": "edit"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/persist.c",
"tokens_estimated": 63,
"action": "edit"
}
],
"totals": {
"input_tokens_estimated": 504,
"output_tokens_estimated": 1974,
"reads_count": 9,
"writes_count": 11,
"repeated_reads_blocked": 5,
"anatomy_lookups": 1
}
} }
], ],
"daemon_usage": [], "daemon_usage": [],

33
src/include/request.h
View file

@ -176,10 +176,12 @@ extern int xce_upgrade_config_req_extract_raw(const uint8_t *pkt_data,
size_t *out_len, size_t *out_len,
uint8_t **out_sha256); uint8_t **out_sha256);
/* ── Task 10.2: Client-Side Transparent Retry (Req 7.1–7.6) ───────────────── /* ── Client-Side Transparent Retry tunables (Req 6.1, 6.2) ──────────────────
* *
* xce_retry_config_t: configurable retry parameters with safe production defaults. * xce_retry_config_t: configurable retry parameters with safe production
* Pass NULL to xce_request_with_retry() to use defaults. * defaults, consumed by the libxce client failover loop (client.c /
* client_failover.c). XCE_RETRY_DEFAULTS supplies the defaults applied when no
* override is configured.
*/ */
typedef struct { typedef struct {
int max_retries; /* default 20 */ int max_retries; /* default 20 */
@ -190,29 +192,4 @@ typedef struct {
extern const xce_retry_config_t XCE_RETRY_DEFAULTS; extern const xce_retry_config_t XCE_RETRY_DEFAULTS;
/*
* xce_request_with_retry: transparent retry wrapper for transient controller errors.
*
* - XCE_ERR_NOT_READY → backoff, retry same endpoint (Req 7.1)
* - XCE_ERR_STANDBY with redirect_host/port → reconnect, retry (Req 7.2)
* - XCE_ERR_STANDBY without redirect → backoff, retry (Req 7.2 fallback)
* - ECONNREFUSED / ETIMEDOUT → reconnect, retry (Req 7.3)
* - EPERM / EINVAL / ENOENT / EACCES → surface immediately (Req 7.4)
* - Exhausted max_retries → surface last error (Req 7.5)
* - Backoff jitter via rand_r() to prevent thundering herd (Req 7.6)
*
* NULL cfg → uses XCE_RETRY_DEFAULTS.
*/
extern int xce_request_with_retry(endpoint_t *ep,
xce_request_t *req,
const xce_retry_config_t *cfg);
/*
* xce_retry_send_hook: test-only injection point for the send+receive round-trip.
* When non-NULL, xce_request_with_retry() calls this instead of the real network path.
* Returns 0 on success (reply_out populated), -errno on transport failure.
*/
extern int (*xce_retry_send_hook)(endpoint_t *ep, xce_request_t *req,
xce_reply_t *reply_out);
#endif #endif

10
src/include/xcectx.h
View file

@ -2,6 +2,9 @@
#define _XCECTX_H_ 1 #define _XCECTX_H_ 1
#include <stdint.h>
#include "config.h" /* XCE_MAXHOSTNAME_LEN */
#include "netio.h" #include "netio.h"
#include "request.h" #include "request.h"
#include "vtaskmgr.h" #include "vtaskmgr.h"
@ -28,6 +31,13 @@ typedef struct {
cbuf_t wbuf; cbuf_t wbuf;
dataset_t* dataset; dataset_t* dataset;
/* Redirect-hint address captured from the last redirect reply by
* handle_reply() (which only sees xce_ctx_t) and read by the failover
* rotator on the next retry. Empty host ("") means no hint. Advisory only;
* correctness never depends on it. */
char redirect_hint_host[XCE_MAXHOSTNAME_LEN];
uint16_t redirect_hint_port;
struct { struct {
union { union {
char* debugmsg; char* debugmsg;

29
src/lib/client.c
View file

@ -4,6 +4,7 @@
#include <sys/socket.h> #include <sys/socket.h>
#include "client.h" #include "client.h"
#include "client_internal.h"
#include "controller.capnp.h" #include "controller.capnp.h"
#include "mylock.h" #include "mylock.h"
@ -13,19 +14,6 @@
#include "xcectx.h" #include "xcectx.h"
#include "xlist.h" #include "xlist.h"
typedef enum {
API_CTX_INIT = 0,
API_CTX_RUN,
API_CTX_DOWN,
API_CTX_ERR
} api_ctx_state_t;
typedef struct {
xce_client_config_t* conf;
endpoint_t ep;
xce_ctx_t ctx;
} api_ctx_t;
int xcecli_ctx_init(xcecli_ctx_t* c, xce_client_config_t* cfg) int xcecli_ctx_init(xcecli_ctx_t* c, xce_client_config_t* cfg)
{ {
api_ctx_t* ctx; api_ctx_t* ctx;
@ -126,6 +114,20 @@ int xcecli_start(xcecli_ctx_t c)
return rc; return rc;
} }
/* Initialize per-context failover state: sticky primary starts at
* masters[0] (the initial connect target above), retry tunables default
* to XCE_RETRY_DEFAULTS, and no rejoin hook is installed for pure clients.
*/
rc = mylock_init(&(ctx->failover.lock));
if (rc < 0) {
endpoint_fini(&(ctx->ep));
return rc;
}
ctx->failover.cur_master = 0;
ctx->failover.retry = XCE_RETRY_DEFAULTS;
ctx->failover.rejoin_cb = NULL;
ctx->failover.rejoin_arg = NULL;
rc = xce_ctx_start(&(ctx->ctx)); rc = xce_ctx_start(&(ctx->ctx));
return rc; return rc;
@ -141,6 +143,7 @@ int xcecli_stop(xcecli_ctx_t c)
xce_ctx_stop(&(ctx->ctx)); xce_ctx_stop(&(ctx->ctx));
endpoint_fini(&(ctx->ep)); endpoint_fini(&(ctx->ep));
mylock_fini(&(ctx->failover.lock));
return 0; return 0;
} }

63
src/lib/client_internal.h Normal file
View file

@ -0,0 +1,63 @@
#if !defined(_CLIENT_INTERNAL_H_)
#define _CLIENT_INTERNAL_H_ 1
#include <stdint.h>
#include "client.h"
#include "mylock.h"
#include "netio.h"
#include "request.h"
#include "xcectx.h"
/*
* client_internal.h — private libxce client state shared between client.c and
* the forthcoming failover helper (client_failover.c, task 1.x).
*
* api_ctx_t was previously file-local to client.c. It is moved here so the
* failover helper can operate on it without exposing it through the public
* API in client.h.
*/
typedef enum {
API_CTX_INIT = 0,
API_CTX_RUN,
API_CTX_DOWN,
API_CTX_ERR
} api_ctx_state_t;
/*
* xcecli_reconnect_cb: consumer rejoin hook invoked after a transparent
* reconnect/failover and before the in-flight request resumes. NULL for pure
* clients (e.g. xce_test). The public declaration and the
* xcecli_set_reconnect_handler() setter are introduced by a later task; this
* private typedef lets the failover state carry the hook now.
*/
typedef int (*xcecli_reconnect_cb)(xcecli_ctx_t c, void* arg);
/*
* xce_failover_state_t: per-context failover state. Persists across calls so a
* sticky primary and the retry tunables survive between requests.
*
* lock — serializes xcecli_* per ctx (shared-ctx safety)
* cur_master — index into conf->masters[] (sticky primary)
* retry — defaults from XCE_RETRY_DEFAULTS unless overridden
* rejoin_cb — consumer rejoin hook; NULL for pure clients
* rejoin_arg — opaque argument passed to rejoin_cb
*/
typedef struct {
mylock_t lock;
int cur_master;
xce_retry_config_t retry;
xcecli_reconnect_cb rejoin_cb;
void* rejoin_arg;
} xce_failover_state_t;
typedef struct {
xce_client_config_t* conf;
endpoint_t ep;
xce_ctx_t ctx;
xce_failover_state_t failover;
} api_ctx_t;
#endif

104
src/lib/request.c
View file

@ -1920,109 +1920,17 @@ int xce_clear_reason_req(endpoint_t* ep, xce_clear_reason_t* reason)
return xce_request(ep, &request); return xce_request(ep, &request);
} }
/* ── Task 10.2: Client-Side Transparent Retry (Req 7.1–7.6) ──────────────── /* ── Client-Side Transparent Retry tunables (Req 6.1, 6.2) ─────────────────
* *
* xce_retry_send_hook: test-injectable send+receive round-trip. * XCE_RETRY_DEFAULTS provides safe production defaults for the failover retry
* NULL in production; set by unit tests to return controlled replies. * loop that lives in the libxce client layer (client.c / client_failover.c).
* The orphaned no-op xce_request_with_retry() primitive and its
* xce_retry_send_hook test seam were removed; they are superseded by the
* xcecli_* failover loop.
*/ */
int (*xce_retry_send_hook)(endpoint_t *ep, xce_request_t *req,
xce_reply_t *reply_out) = NULL;
const xce_retry_config_t XCE_RETRY_DEFAULTS = { const xce_retry_config_t XCE_RETRY_DEFAULTS = {
.max_retries = 20, .max_retries = 20,
.initial_retry_ms = 50, .initial_retry_ms = 50,
.max_retry_ms = 5000, .max_retry_ms = 5000,
.backoff_multiplier = 2.0, .backoff_multiplier = 2.0,
}; };
/* is_non_retriable: errors that should surface immediately without retry */
static int is_non_retriable(int status)
{
return (status == -EPERM || status == -EACCES ||
status == -EINVAL || status == -ENOENT);
}
/* ms_sleep_jittered: sleep delay_ms + random jitter via rand_r (Req 7.6) */
static void ms_sleep_jittered(int delay_ms, unsigned int *seed)
{
if (delay_ms <= 0) return;
int jitter_ms = (int)((rand_r(seed) % (delay_ms / 4 + 1)));
long total_ns = ((long)delay_ms + jitter_ms) * 1000000L;
struct timespec ts = { .tv_sec = total_ns / 1000000000L,
.tv_nsec = total_ns % 1000000000L };
nanosleep(&ts, NULL);
}
int xce_request_with_retry(endpoint_t *ep,
xce_request_t *req,
const xce_retry_config_t *cfg)
{
if (cfg == NULL)
cfg = &XCE_RETRY_DEFAULTS;
unsigned int rand_seed = (unsigned int)(uintptr_t)ep ^ (unsigned int)time(NULL);
int delay_ms = cfg->initial_retry_ms;
int last_rc = -ETIMEDOUT;
for (int attempt = 0; attempt <= cfg->max_retries; attempt++) {
xce_reply_t reply;
memset(&reply, 0, sizeof(reply));
int rc;
if (xce_retry_send_hook) {
rc = xce_retry_send_hook(ep, req, &reply);
} else {
/* Production path: fire-and-forget send; reply handling is caller's
* responsibility via the event loop. Return send error directly. */
rc = xce_request(ep, req);
if (rc != 0)
return rc;
return 0; /* async: caller reads reply via event loop */
}
/* Transport-level failure (ECONNREFUSED, ETIMEDOUT) → retry (Req 7.3) */
if (rc == -ECONNREFUSED || rc == -ETIMEDOUT) {
last_rc = rc;
if (attempt < cfg->max_retries)
ms_sleep_jittered(delay_ms, &rand_seed);
goto next_backoff;
}
/* Any other transport error → surface immediately */
if (rc != 0)
return rc;
/* Application-layer status from reply */
int status = (int)reply.status;
if (status == 0)
return 0; /* success */
last_rc = status;
/* Non-retriable errors: surface immediately (Req 7.4) */
if (is_non_retriable(status))
return status;
/* XCE_ERR_STANDBY with redirect → reconnect (Req 7.2).
* In hook mode, the hook already updated ep/redirect state.
* Just retry; redirect handling is at the transport layer. */
if (status == -E_XCE_STANDBY && reply.redirect_host != NULL
&& reply.redirect_port != 0) {
/* no extra sleep for redirect — just retry immediately */
continue;
}
/* XCE_ERR_NOT_READY or XCE_ERR_STANDBY without redirect → backoff */
if (attempt < cfg->max_retries)
ms_sleep_jittered(delay_ms, &rand_seed);
next_backoff:
{
int next = (int)((double)delay_ms * cfg->backoff_multiplier);
delay_ms = next > cfg->max_retry_ms ? cfg->max_retry_ms : next;
}
}
return last_rc;
}

336
src/lib/testsuite/test_libxce_retry.c
View file

@ -1,324 +1,35 @@
/* /*
* test_libxce_retry.c — RED-phase tests for xce_request_with_retry() (tasks 10.1/10.2). * test_libxce_retry.c — placeholder for the libxce failover retry tests.
* *
* Uses xce_retry_send_hook to inject mock round-trip outcomes without real * The orphaned xce_request_with_retry() primitive and its xce_retry_send_hook
* network connections. Each mock call returns a configured status/redirect. * test seam were removed (superseded by the xcecli_* failover loop). Task 1.2
* rewrites this suite against the new failover model (StatusClassifier, master
* rotation, redirect-hint, budget → cluster-not-ready).
* *
* Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6 * For now this is a minimal compiling placeholder that asserts the retained
* retry-tunables scaffolding (xce_retry_config_t + XCE_RETRY_DEFAULTS) is
* reachable, keeping the lib test suite GREEN between tasks 1.1 and 1.2.
*
* Requirements: 6.1, 6.2
*/ */
#define _GNU_SOURCE #define _GNU_SOURCE
#include <errno.h>
#include <pthread.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <stdio.h> #include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <unity/unity.h> #include <unity/unity.h>
#include "request.h" #include "request.h"
#include "xce/errnox.h"
/* ── Forward declarations for functions-under-test ───────────────── */ /* Req 6.1, 6.2: the retry-tunables struct + safe production defaults are
/* xce_request_with_retry is not yet implemented — linker will fail (RED). */ * retained and reachable for the failover loop. */
extern int xce_request_with_retry(endpoint_t *ep, static void test_retry_defaults_are_reachable(void)
xce_request_t *req,
const xce_retry_config_t *cfg);
/*
* xce_retry_send_hook: test seam for synchronous request-reply.
* When set, xce_request_with_retry calls this instead of the real network path.
* The hook fills in *reply_out and returns 0 on success, -errno on failure.
*/
extern int (*xce_retry_send_hook)(endpoint_t *ep, xce_request_t *req,
xce_reply_t *reply_out);
/* ── Default retry config for tests (fast backoff) ──────────────── */
static const xce_retry_config_t g_fast_cfg = {
.max_retries = 5,
.initial_retry_ms = 1,
.max_retry_ms = 10,
.backoff_multiplier = 2.0,
};
/* ── Mock state ─────────────────────────────────────────────────── */
#define MAX_MOCK_REPLIES 32
struct mock_reply {
int32_t status; /* xce_reply_t.status to return */
int send_rc; /* return code from the hook (0 or -errno) */
char *redirect_host; /* non-NULL if status == -E_XCE_STANDBY */
uint16_t redirect_port;
};
static struct mock_reply g_replies[MAX_MOCK_REPLIES];
static int g_reply_count = 0;
static int g_reply_idx = 0;
static int g_call_count = 0;
static void reset_mock(void)
{ {
for (int i = 0; i < g_reply_count; i++) { xce_retry_config_t cfg = XCE_RETRY_DEFAULTS;
free(g_replies[i].redirect_host);
g_replies[i].redirect_host = NULL;
}
g_reply_count = 0;
g_reply_idx = 0;
g_call_count = 0;
memset(g_replies, 0, sizeof(g_replies));
}
static void add_reply(int32_t status, int send_rc, TEST_ASSERT_EQUAL_INT(20, cfg.max_retries);
const char *rhost, uint16_t rport) TEST_ASSERT_EQUAL_INT(50, cfg.initial_retry_ms);
{ TEST_ASSERT_EQUAL_INT(5000, cfg.max_retry_ms);
if (g_reply_count >= MAX_MOCK_REPLIES) return; TEST_ASSERT_TRUE(cfg.backoff_multiplier > 1.0);
int i = g_reply_count++;
g_replies[i].status = status;
g_replies[i].send_rc = send_rc;
g_replies[i].redirect_host = rhost ? strdup(rhost) : NULL;
g_replies[i].redirect_port = rport;
}
static int mock_send(endpoint_t *ep, xce_request_t *req, xce_reply_t *reply_out) printf("test_retry_defaults_are_reachable: PASS\n");
{
(void)ep; (void)req;
g_call_count++;
if (g_reply_idx >= g_reply_count) {
/* Default: return success after mocks exhausted */
memset(reply_out, 0, sizeof(*reply_out));
reply_out->status = 0;
return 0;
}
struct mock_reply *m = &g_replies[g_reply_idx++];
if (m->send_rc != 0)
return m->send_rc;
memset(reply_out, 0, sizeof(*reply_out));
reply_out->status = m->status;
reply_out->redirect_host = m->redirect_host;
reply_out->redirect_port = m->redirect_port;
return 0;
}
static void install_mock(void)
{
xce_retry_send_hook = mock_send;
}
/* ── Tests ──────────────────────────────────────────────────────── */
/* Req 7.1: E_XCE_NOT_READY → retry with backoff, eventually succeed */
static void test_not_ready_retries_then_succeeds(void)
{
install_mock();
reset_mock();
/* 3 NOT_READY responses, then success */
add_reply(-E_XCE_NOT_READY, 0, NULL, 0);
add_reply(-E_XCE_NOT_READY, 0, NULL, 0);
add_reply(-E_XCE_NOT_READY, 0, NULL, 0);
add_reply(0, 0, NULL, 0); /* success */
xce_request_t req;
memset(&req, 0, sizeof(req));
int rc = xce_request_with_retry((endpoint_t*)0x1, &req, &g_fast_cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(4, g_call_count);
printf("test_not_ready_retries_then_succeeds: PASS\n");
}
/* Req 7.2: E_XCE_STANDBY with redirect → reconnect to redirect, retry */
static void test_standby_with_redirect_reconnects_and_retries(void)
{
install_mock();
reset_mock();
/* Standby response with redirect */
add_reply(-E_XCE_STANDBY, 0, "primary.host", 9990);
add_reply(0, 0, NULL, 0); /* success on redirect target */
xce_request_t req;
memset(&req, 0, sizeof(req));
int rc = xce_request_with_retry((endpoint_t*)0x1, &req, &g_fast_cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(2, g_call_count);
printf("test_standby_with_redirect_reconnects_and_retries: PASS\n");
}
/* Req 7.2 fallback: E_XCE_STANDBY without redirect → backoff, retry same */
static void test_standby_without_redirect_falls_back_to_backoff(void)
{
install_mock();
reset_mock();
/* STANDBY without redirect, then success */
add_reply(-E_XCE_STANDBY, 0, NULL, 0);
add_reply(0, 0, NULL, 0);
xce_request_t req;
memset(&req, 0, sizeof(req));
int rc = xce_request_with_retry((endpoint_t*)0x1, &req, &g_fast_cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(2, g_call_count);
printf("test_standby_without_redirect_falls_back_to_backoff: PASS\n");
}
/* Req 7.4: non-retriable error (EPERM) → surface immediately, no retry */
static void test_non_retriable_error_surfaces_immediately(void)
{
install_mock();
reset_mock();
/* Auth/perm failure — must NOT retry */
add_reply(-EPERM, 0, NULL, 0);
/* Add a success after — it should never be reached */
add_reply(0, 0, NULL, 0);
xce_request_t req;
memset(&req, 0, sizeof(req));
int rc = xce_request_with_retry((endpoint_t*)0x1, &req, &g_fast_cfg);
TEST_ASSERT_EQUAL_INT(-EPERM, rc);
TEST_ASSERT_EQUAL_INT(1, g_call_count); /* only 1 attempt */
printf("test_non_retriable_error_surfaces_immediately: PASS\n");
}
/* Req 7.5: max_retries exhausted → surface last error */
static void test_max_retries_exhausted_returns_last_error(void)
{
install_mock();
reset_mock();
/* Fill with NOT_READY beyond max_retries */
for (int i = 0; i <= g_fast_cfg.max_retries + 1; i++)
add_reply(-E_XCE_NOT_READY, 0, NULL, 0);
xce_request_t req;
memset(&req, 0, sizeof(req));
int rc = xce_request_with_retry((endpoint_t*)0x1, &req, &g_fast_cfg);
/* Must return the last error, not 0 */
TEST_ASSERT_NOT_EQUAL(0, rc);
/* Must have tried max_retries+1 times total */
TEST_ASSERT_EQUAL_INT(g_fast_cfg.max_retries + 1, g_call_count);
printf("test_max_retries_exhausted_returns_last_error: PASS\n");
}
/* Req 7.3: ECONNREFUSED → reconnect, retry */
static void test_conn_refused_retries(void)
{
install_mock();
reset_mock();
add_reply(0, -ECONNREFUSED, NULL, 0);
add_reply(0, 0, NULL, 0);
xce_request_t req;
memset(&req, 0, sizeof(req));
int rc = xce_request_with_retry((endpoint_t*)0x1, &req, &g_fast_cfg);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(2, g_call_count);
printf("test_conn_refused_retries: PASS\n");
}
/* NULL cfg → uses defaults (max_retries=20, etc.) */
static void test_null_cfg_uses_defaults(void)
{
install_mock();
reset_mock();
add_reply(0, 0, NULL, 0); /* immediate success */
xce_request_t req;
memset(&req, 0, sizeof(req));
int rc = xce_request_with_retry((endpoint_t*)0x1, &req, NULL);
TEST_ASSERT_EQUAL_INT(0, rc);
printf("test_null_cfg_uses_defaults: PASS\n");
}
/* Req 7.6: 50 concurrent simulated clients all succeed (backoff prevents starvation) */
struct concurrent_client_args {
int id;
int rc;
atomic_int *success_count;
};
static void *concurrent_client_fn(void *arg)
{
struct concurrent_client_args *a = arg;
xce_request_t req;
memset(&req, 0, sizeof(req));
/* Each client sees NOT_READY twice, then success */
int rc = xce_request_with_retry((endpoint_t*)0x1, &req, &g_fast_cfg);
a->rc = rc;
if (rc == 0)
atomic_fetch_add(a->success_count, 1);
return NULL;
}
static int concurrent_mock_send(endpoint_t *ep, xce_request_t *req,
xce_reply_t *reply_out)
{
(void)ep; (void)req;
static atomic_int s_call_count = 0;
int n = atomic_fetch_add(&s_call_count, 1);
memset(reply_out, 0, sizeof(*reply_out));
/* First 3 calls per "client" return NOT_READY, then succeed.
* Since we can't track per-client state here, use a simple global
* threshold: first 100 calls NOT_READY, rest succeed. */
if (n < 100)
reply_out->status = -E_XCE_NOT_READY;
else
reply_out->status = 0;
return 0;
}
static void test_50_concurrent_clients_all_succeed(void)
{
xce_retry_send_hook = concurrent_mock_send;
const int N = 50;
pthread_t threads[50];
struct concurrent_client_args args[50];
atomic_int success_count;
atomic_init(&success_count, 0);
for (int i = 0; i < N; i++) {
args[i].id = i;
args[i].rc = -1;
args[i].success_count = &success_count;
pthread_create(&threads[i], NULL, concurrent_client_fn, &args[i]);
}
/* Wait for all clients (with 30s overall deadline from Unity timeout) */
for (int i = 0; i < N; i++)
pthread_join(threads[i], NULL);
TEST_ASSERT_EQUAL_INT(N, atomic_load(&success_count));
printf("test_50_concurrent_clients_all_succeed: PASS (%d/%d)\n",
atomic_load(&success_count), N);
} }
/* ── Suite entry point ──────────────────────────────────────────── */ /* ── Suite entry point ──────────────────────────────────────────── */
@ -326,13 +37,6 @@ static void test_50_concurrent_clients_all_succeed(void)
void test_libxce_retry(void) void test_libxce_retry(void)
{ {
UNITY_BEGIN(); UNITY_BEGIN();
RUN_TEST(test_not_ready_retries_then_succeeds); RUN_TEST(test_retry_defaults_are_reachable);
RUN_TEST(test_standby_with_redirect_reconnects_and_retries);
RUN_TEST(test_standby_without_redirect_falls_back_to_backoff);
RUN_TEST(test_non_retriable_error_surfaces_immediately);
RUN_TEST(test_max_retries_exhausted_returns_last_error);
RUN_TEST(test_conn_refused_retries);
RUN_TEST(test_null_cfg_uses_defaults);
RUN_TEST(test_50_concurrent_clients_all_succeed);
UNITY_END(); UNITY_END();
} }