commit missed
This commit is contained in:
parent
a8840dbba5
commit
c45f80ab2b
15 changed files with 14109 additions and 1268 deletions
332
.kiro/specs/ha-prevote/design.md
Normal file
332
.kiro/specs/ha-prevote/design.md
Normal file
|
|
@ -0,0 +1,332 @@
|
|||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** This feature adds a **PreVote phase** to the XCE controller HA election path so that a controller which is merely temporarily unreachable cannot inflate its persistent term or unseat a healthy primary. Before a node increments its term and campaigns, it must first win a hypothetical PreVote majority — a "would you vote for me in term+1?" round that mutates no node's persistent term or recorded vote.
|
||||
|
||||
**Users.** Operators of 3-node controller HA clusters, and the controllers themselves, benefit: a frozen-then-resumed standby rejoins as a no-op instead of triggering the ~30s demote-and-restart re-election storm observed live on 2026-06-15.
|
||||
|
||||
**Impact.** The election trigger path changes from `trigger → real election (term++)` to `trigger → PreVote round → (only on majority) → real election (term++)`. The real-vote state machine, replication, and commit engine are unchanged. The change is additive: two new wire opcodes reusing the existing vote payload structs, pure PreVote decision **and tally** functions in the raft library (mirroring the production real-vote path `raft_on_vote_reply`), and PreVote orchestration in the controller that reuses the existing eligibility gate.
|
||||
|
||||
### Goals
|
||||
- A node never increments its persistent term unless a PreVote majority is first obtained.
|
||||
- A voter denies a PreVote whenever it still believes a leader is alive (leader stickiness), reusing the existing heartbeat-freshness predicate.
|
||||
- A frozen-then-resumed standby in a healthy cluster gets no PreVote majority and re-attaches as a standby with the cluster term unchanged.
|
||||
- Genuine leadership changes (true primary death, partition heal) still elect within existing timeout bounds.
|
||||
- All Raft safety invariants and the bug-339/bug-340 fixes are preserved.
|
||||
|
||||
### Non-Goals
|
||||
- Changes to replication, the commit/barrier engine, snapshot/WAL catch-up transfer, or config bootstrap.
|
||||
- Redesign of the freshness / log-completeness comparison (reused as-is).
|
||||
- Backward/rolling-upgrade compatibility (product is pre-release; all controllers run the same build).
|
||||
- Leader-side CheckQuorum self-demotion and the defense-in-depth real-vote stickiness guard (see Architecture → Deferred decisions).
|
||||
- **The startup / first-boot election** (the `ha_elect_ctx_t.done` semaphore path). PreVote gates only the **failover** trigger; on first boot there is no leader to protect and every node's heartbeat is stale, so a PreVote round would trivially pass. The startup election path is untouched.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
- The **PreVote round** on the election-trigger path: a side-effect-free majority probe that runs before any persistent term increment.
|
||||
- The **voter-side PreVote decision**: term check + reuse of the existing eligibility gate + reuse of the dispatcher freshness gate, with **zero persistent mutation**.
|
||||
- The two new wire opcodes (`preVoteRequest`, `preVoteReply`) and their dispatch.
|
||||
- The trigger-path restructuring so that `truncate-WAL + CTRL_ELECTING` happens only after PreVote succeeds.
|
||||
- PreVote observability (denial reasons, abandoned-campaign logging).
|
||||
|
||||
### Out of Boundary
|
||||
- The real-vote state machine (`raft_on_vote_request`, `raft_on_vote_reply`, `raft_start_election`) — invoked unchanged after PreVote success.
|
||||
- The freshness floor computation (`ha_vote_self_freshness`) and the eligibility gate (`raft_cb_vote_eligible`) — reused, not modified (except that the eligibility gate is now also called from the PreVote receiver path).
|
||||
- Replication, commit barrier, snapshot transfer, demotion-convergence mechanics (truncate-above-committed + self-restart) — unchanged.
|
||||
- Leader self-demotion / CheckQuorum.
|
||||
- The startup-election `ha_elect_ctx_t.done` semaphore path — not gated by PreVote.
|
||||
|
||||
### Allowed Dependencies
|
||||
- raft library callback seams (`raft_callbacks_t`), the raft-library vote-tally pattern (`raft_on_vote_reply` ⇒ `raft_on_pre_vote_reply`), `ha_heartbeat_is_fresh`, `ha_vote_self_freshness`, the capnp election transport (`xce_request` / vote endpoints / `peer_ctrl_msg_handler`), and the `xce_vote_request_hook` test seam.
|
||||
- Dependency direction (must not be violated): **capnp/types → raft library (pure) → controller adapter (ha.c) → dispatch/transport (cluster.c)**. The raft library must not depend on controller state.
|
||||
|
||||
### Revalidation Triggers
|
||||
- Any change to the `request_vote_req_t` / `request_vote_reply_t` payload shape (shared by real-vote and PreVote).
|
||||
- Any change to `raft_cb_vote_eligible` semantics (now shared by both vote paths).
|
||||
- Any change to the leader-stickiness predicate (`ha_heartbeat_is_fresh` / `heartbeat_timeout_ms`).
|
||||
- Any change to the election-trigger entry points (`ha_on_primary_ctrl_fd_rdhup`, `liveness_poll_fn`).
|
||||
- Introduction of mixed-version clusters (would reintroduce the deferred wire-compat concern).
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
The election path is split across two layers (preserved by this design):
|
||||
|
||||
- **raft library** (`src/lib/raft/raft.c`, `raft.h`) — the canonical, TLA+-checkable state machine. Owns term, `voted_for`, role, and the pure vote-decision logic. Communicates outward only through `raft_callbacks_t` (no controller dependency). Today: `raft_start_election` increments+persists the term before any RPC; `raft_on_vote_request` adopts a higher term and steps down a leader in **Step 2 — before** the `on_vote_eligible` gate runs. This ordering is exactly why the existing heartbeat-stickiness gate cannot protect a leader from a fresh, higher-term RequestVote, and why the fix must be sender-side.
|
||||
- **controller adapter** (`src/controller/ha.c`, `cluster.c`) — async transport + collection. `cluster.c` dispatches `OPCode_requestVote`, applies the freshness floor, maps candidate name→index, then calls the library. `ha.c` orchestrates `ha_start_election_async` (allocates `ha_elect_ctx_t`, self-vote, delegates to `raft_start_election`), collects replies (`handle_request_vote_reply` → `ha_elect_evaluate_and_signal`), and gates eligibility (`raft_cb_vote_eligible`: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny). Triggers: `ha_on_primary_ctrl_fd_rdhup` (RDHUP) and `liveness_poll_fn` (heartbeat timeout), which today transition to `CTRL_ELECTING`, truncate WAL, and start the election.
|
||||
|
||||
### Architecture Pattern & Boundary Map
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph trigger [Election Trigger]
|
||||
Rdhup[ha_on_primary_ctrl_fd_rdhup]
|
||||
Liveness[liveness_poll_fn]
|
||||
end
|
||||
subgraph controller [Controller Adapter ha.c cluster.c]
|
||||
BeginFailover[ha_begin_failover]
|
||||
PrevoteAsync[ha_start_prevote_async]
|
||||
ElectAsync[ha_start_election_async]
|
||||
Collect[raft_on_pre_vote_reply tally]
|
||||
DispReq[cluster.c preVoteRequest case]
|
||||
DispRep[cluster.c preVoteReply case]
|
||||
Eligible[raft_cb_vote_eligible reused]
|
||||
Fresh[ha_vote_self_freshness reused]
|
||||
end
|
||||
subgraph raftlib [Raft Library pure]
|
||||
Begin[raft_begin_pre_vote]
|
||||
OnReq[raft_on_pre_vote_request]
|
||||
OnRep[raft_on_pre_vote_reply]
|
||||
StartElection[raft_start_election unchanged]
|
||||
end
|
||||
subgraph wire [Transport]
|
||||
Capnp[preVoteRequest preVoteReply opcodes]
|
||||
end
|
||||
|
||||
Rdhup --> BeginFailover
|
||||
Liveness --> BeginFailover
|
||||
BeginFailover --> PrevoteAsync
|
||||
PrevoteAsync --> Begin
|
||||
Begin --> Capnp
|
||||
Capnp --> DispReq
|
||||
DispReq --> Fresh
|
||||
DispReq --> Eligible
|
||||
DispReq --> OnReq
|
||||
OnReq --> Capnp
|
||||
Capnp --> DispRep
|
||||
DispRep --> OnRep
|
||||
OnRep --> Collect
|
||||
Collect -->|majority| ElectAsync
|
||||
ElectAsync --> StartElection
|
||||
Collect -->|lost or timeout| Rearm[re-arm liveness stay STANDBY]
|
||||
```
|
||||
|
||||
**Architecture Integration**
|
||||
- **Selected pattern**: side-effect-free probe interposed before an existing state transition (Raft dissertation §9.6 PreVote). Adopted, not invented.
|
||||
- **Boundaries**: raft library owns the pure PreVote *decision and tally*; the controller owns the *orchestration/transport*. The voter eligibility gate is **shared** with the real-vote path via reuse, not duplication; the PreVote tally mirrors the production real-vote tally (`raft_on_vote_reply`) rather than the test-only `ha_elect_ctx_t` path.
|
||||
- **Existing patterns preserved**: callback-seam isolation of the raft library; raft-library-owns-tally (`raft_on_vote_reply` ⇒ `raft_on_pre_vote_reply`); dispatcher-applies-freshness-before-library; `xce_vote_request_hook` test seam.
|
||||
- **New components rationale**: a new opcode pair keeps the real-vote handler byte-for-byte unchanged (zero regression risk to bug-339/340); pure PreVote functions in the library keep the state machine TLA+-checkable.
|
||||
- **Steering compliance**: async, non-blocking vtask handlers; long-lived connections; inbound connections vote-only; immutable-term discipline (PreVote mutates nothing).
|
||||
|
||||
### Deferred decisions (documented, not implemented)
|
||||
- **Defense-in-depth real-vote stickiness guard** — rejecting a real higher-term RequestVote while a leader is believed alive. PreVote already guarantees a candidate only sends a real RequestVote after a majority's stickiness has lapsed, so in a same-version cluster no stickiness-violating real RequestVote can arise. Adding the guard would touch the safety-critical real-vote term-adoption path and risk regressing bug-339/340 for no in-scope benefit. Deferred; revisit only if mixed-version clusters or leader CheckQuorum are introduced.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice / Version | Role in Feature | Notes |
|
||||
|-------|------------------|-----------------|-------|
|
||||
| Messaging / Events | Cap'n Proto (capnpc-c 1.0.1) | New `preVoteRequest`/`preVoteReply` opcodes reusing existing vote payload structs | No new structs; follows existing opcode-enum pattern |
|
||||
| Backend / Services | C (raft library + controller) | Pure PreVote decision + tally functions; controller orchestration | No new libraries; reuses iomgr/delaymgr and the raft tally pattern |
|
||||
| Infrastructure / Runtime | iomgr async transport, delaymgr | PreVote send/collect + round timeout | Non-blocking, mirrors election path |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Modified Files
|
||||
- `src/schema/controller.capnp` — add two `OPCode` enum entries: `preVoteRequest`, `preVoteReply`. Reuse existing `RequestVoteRequest` / `RequestVoteReply` structs and their `Request.body` union arms (`requestvotemsg` / `requestvotereplymsg`); no new payload structs.
|
||||
- `src/lib/raft/raft.h` — add pre-vote return codes (e.g. `RAFT_PRE_VOTE_WON`, `RAFT_PRE_VOTE_LOST`, `RAFT_PRE_VOTE_PENDING`), an optional `on_send_pre_vote_request` callback in `raft_callbacks_t`, pre-vote tally fields in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`), and declarations for the three new functions.
|
||||
- `src/lib/raft/raft.c` — add `raft_begin_pre_vote`, `raft_on_pre_vote_request`, `raft_on_pre_vote_reply` (all pure: no `on_persist`, no `term`/`voted_for`/`role` mutation). `raft_on_pre_vote_request` reuses `on_vote_eligible`. **`raft_on_pre_vote_reply` owns the PreVote tally in `raft_state_t` (mirroring `raft_on_vote_reply`) and returns `WON/LOST/PENDING` — it never fires `on_become_leader`/`on_become_follower`.**
|
||||
- `src/controller/ha.h` — add a controller-side PreVote round handle on `ha_state_t` (round-timeout context + in-progress flag for cancellation). **No change to `ha_elect_ctx_t`** (the tally lives in `raft_state_t`, not here).
|
||||
- `src/controller/ha.c` — add `ha_begin_failover` (common entry), `ha_start_prevote_async` (calls `raft_begin_pre_vote`, arms round timeout), and `pre_vote_reply_dispatch` (calls `raft_on_pre_vote_reply`, mirroring `vote_reply_dispatch → raft_on_vote_reply`); on `RAFT_PRE_VOTE_WON` perform truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST`/round-timeout stay `CTRL_STANDBY`, re-arm liveness, log abandonment. Modify `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn` to call `ha_begin_failover` (PreVote first; **no** truncate/`CTRL_ELECTING` until PreVote wins); modify `ha_on_heartbeat_received` to cancel an in-progress PreVote and re-attach.
|
||||
- `src/controller/cluster.c` — add `OPCode_preVoteRequest` and `OPCode_preVoteReply` dispatch cases mirroring the real-vote cases, calling `raft_on_pre_vote_request` (no term adoption) and routing replies to `pre_vote_reply_dispatch`.
|
||||
|
||||
### New Files
|
||||
- `src/controller/testsuite/test_ha_prevote.c` — PreVote gating + frozen-then-resumed no-op scenario + genuine-failover scenario.
|
||||
|
||||
### Test files extended
|
||||
- `src/lib/testsuite/test_raft.c` — pure PreVote decision/tally unit tests + invariant assertions (no term mutation).
|
||||
- `src/controller/testsuite/test_ha_vote_gate.c` — PreVote eligibility reuse of the role/heartbeat/catch-up gates.
|
||||
|
||||
## System Flows
|
||||
|
||||
### PreVote round (candidate side)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Trig as Trigger
|
||||
participant Cand as Candidate ha.c
|
||||
participant Peer as Peer voter
|
||||
Trig->>Cand: ha_begin_failover (stay CTRL_STANDBY)
|
||||
Cand->>Cand: ha_start_prevote_async (prospective term = current+1, no persist)
|
||||
Cand->>Peer: preVoteRequest term+1 tag lsn
|
||||
Note over Peer: stickiness gate then freshness gate then term check
|
||||
Peer-->>Cand: preVoteReply granted or denied (peer term unchanged)
|
||||
alt PreVote majority
|
||||
Cand->>Cand: now truncate WAL, CTRL_ELECTING
|
||||
Cand->>Cand: ha_start_election_async then raft_start_election (term++)
|
||||
else lost or round timeout
|
||||
Cand->>Cand: stay CTRL_STANDBY, re-arm liveness watchdog
|
||||
Note over Cand: next primary heartbeat re-attaches as standby
|
||||
end
|
||||
```
|
||||
|
||||
Key decisions: the candidate remains `CTRL_STANDBY` and mutates nothing during the PreVote round; the destructive `truncate + CTRL_ELECTING` step is deferred to PreVote success. An incoming primary heartbeat during the round cancels it and re-attaches. The round is bounded by `election_timeout_ms`; on loss the watchdog is re-armed so a later genuine failure still triggers, and repeated PreVote attempts succeed once a majority's stickiness lapses.
|
||||
|
||||
### PreVote decision (voter side)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[preVoteRequest received] --> B{role == CTRL_PRIMARY}
|
||||
B -- yes --> D[deny]
|
||||
B -- no --> C{role == CTRL_STANDBY and heartbeat fresh}
|
||||
C -- yes --> D
|
||||
C -- no --> E{in catch-up}
|
||||
E -- yes --> D
|
||||
E -- no --> F{req.term lt current_term}
|
||||
F -- yes --> D
|
||||
F -- no --> G{freshness floor ok}
|
||||
G -- no --> D
|
||||
G -- yes --> H[grant, no term change, no persist, no voted_for]
|
||||
```
|
||||
|
||||
Steps B/C/E are exactly the existing `raft_cb_vote_eligible` gates (reused). Step G is exactly the existing dispatcher freshness floor (`ha_vote_self_freshness`). The only PreVote-specific rule is the **absence of any state mutation** on either grant or deny.
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Interfaces / Flows |
|
||||
|-------------|---------|------------|--------------------|
|
||||
| 1.1 | PreVote before term increment on trigger | `ha_begin_failover`, `ha_start_prevote_async` | PreVote round flow |
|
||||
| 1.2 | No term/vote mutation during round | `raft_begin_pre_vote`, `ha_start_prevote_async` | candidate stays STANDBY |
|
||||
| 1.3 | Request prospective term+1, not persisted | `raft_begin_pre_vote` | preVoteRequest payload |
|
||||
| 1.4 | No majority → no real election | `raft_on_pre_vote_reply` (LOST), `pre_vote_reply_dispatch` | round flow "lost" path |
|
||||
| 1.5 | Majority → start real election | `ha_start_election_async`, `raft_start_election` (unchanged) | round flow "won" path |
|
||||
| 2.1 | Grant only if real-vote freshness gate passes | `cluster.c preVoteRequest`, `ha_vote_self_freshness` | voter flow step G |
|
||||
| 2.2 | Fail freshness → deny | same | voter flow step G |
|
||||
| 2.3 | Grant/deny records no vote, no term | `raft_on_pre_vote_request` | voter flow leaf H |
|
||||
| 2.4 | PreVote does not consume the real vote | `raft_on_pre_vote_request` (no `voted_for` write) | — |
|
||||
| 3.1 | Deny while leader believed alive | `raft_cb_vote_eligible` (`ha_heartbeat_is_fresh`) | voter flow step C |
|
||||
| 3.2 | Past self-election threshold → freshness-only eval | `raft_cb_vote_eligible` | voter flow C→E→F→G |
|
||||
| 3.3 | Primary denies PreVotes | `raft_cb_vote_eligible` (PRIMARY-deny) | voter flow step B |
|
||||
| 4.1 | Frozen-resumed gets no majority | full path | frozen-rejoin scenario test |
|
||||
| 4.2 | Re-attach on next heartbeat, no term change | `ha_on_heartbeat_received` | round flow "lost"/cancel |
|
||||
| 4.3 | Healthy primary not demoted/restarted | PreVote denies → no real RequestVote | — |
|
||||
| 4.4 | Term unchanged, no re-election cycle | candidate stays STANDBY | — |
|
||||
| 5.1 | True death → PreVotes granted → elect | `raft_cb_vote_eligible` (stale heartbeat → eligible) | genuine-failover test |
|
||||
| 5.2 | Partition heal → adopt higher term | `raft_on_vote_reply` (unchanged) | real-vote path |
|
||||
| 5.3 | No regression of bug-339/bug-340 | real-vote path untouched | — |
|
||||
| 5.4 | Complete within election-timeout bounds | `election_timeout_ms` round timeout | round flow |
|
||||
| 6.1 | Election Safety preserved | real-vote path untouched; PreVote grants no leadership | RN-1 (research.md) |
|
||||
| 6.2 | Term monotonicity; PreVote never changes term | `raft_on_pre_vote_*` (pure) | — |
|
||||
| 6.3 | Leader Completeness, monotonic commit index | freshness floor reused; commit path untouched | — |
|
||||
| 6.4 | Consistent with raft.tla + checklist | `test_raft.c` invariant tests | RN-1 |
|
||||
| 7.1 | PreVote over existing transport, no disruption to real votes | new opcodes, reused arms | — |
|
||||
| 7.2 | Log PreVote denials with candidate + reason | `cluster.c preVoteRequest`, `on_trace`/`tlog` | — |
|
||||
| 7.3 | Log abandoned campaign | `pre_vote_reply_dispatch` (LOST/timeout) | round flow |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|
||||
|-----------|-------|--------|--------------|--------------------------|-----------|
|
||||
| raft pre-vote functions | raft library | Pure PreVote decision + tally | 1.2, 1.3, 2.3, 2.4, 6.2 | `on_vote_eligible` (P0), `on_send_pre_vote_request` (P1) | Service, State |
|
||||
| PreVote dispatch | cluster.c | Receive/answer PreVote, no mutation | 2.1, 2.2, 3.1–3.3, 7.1, 7.2 | `ha_vote_self_freshness` (P0), `raft_on_pre_vote_request` (P0) | Service, Event |
|
||||
| PreVote orchestration | ha.c | Run round, gate trigger, branch to election | 1.1, 1.4, 1.5, 4.x, 5.4, 7.3 | `raft_on_pre_vote_reply` (P0), `ha_start_election_async` (P0), `liveness` (P1) | Service, State |
|
||||
| Trigger restructuring | ha.c | Defer truncate/CTRL_ELECTING until PreVote wins | 1.1, 4.2, 4.3, 4.4 | `ha_on_primary_ctrl_fd_rdhup`, `liveness_poll_fn` (P0) | State |
|
||||
|
||||
### Raft library — pre-vote functions
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Pure PreVote decision and tally; canonical, TLA+-checkable; mutates no persistent state |
|
||||
| Requirements | 1.2, 1.3, 2.3, 2.4, 6.2 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `raft_begin_pre_vote`: compute prospective term = `term + 1` **without** assigning it to `s->term` or persisting; set `pre_vote_in_progress`, `pre_votes_granted = 1` (self); emit `on_send_pre_vote_request(ctx, peer_idx, prospective_term)` per peer.
|
||||
- `raft_on_pre_vote_request`: deny if `cand_term < s->term`; else return the result of `on_vote_eligible(ctx, from_idx, cand_term)`; **never** touches `term`, `voted_for`, `role`, or calls `on_persist`. Freshness is applied by the dispatcher before this call (same contract as real vote).
|
||||
- `raft_on_pre_vote_reply`: **owns the PreVote tally** in `raft_state_t` (`pre_votes_granted`, `pre_vote_replies`), mirroring `raft_on_vote_reply`; return `RAFT_PRE_VOTE_WON` at quorum, `RAFT_PRE_VOTE_LOST` when all replied without quorum, else `RAFT_PRE_VOTE_PENDING`. Unlike `raft_on_vote_reply`, it **never** fires `on_become_leader`/`on_become_follower` and makes no role/term change — the controller acts on the returned status.
|
||||
- **Invariant**: across all three functions, `s->term`, `s->voted_for`, `s->voted_for_term`, and `s->role` are unchanged, and `on_persist` is never invoked.
|
||||
|
||||
**Contracts**: Service [x] / State [x]
|
||||
|
||||
##### Service Interface
|
||||
```c
|
||||
/* All pure: no persist, no term/role/voted_for mutation. */
|
||||
int raft_begin_pre_vote(raft_state_t *s); /* sends pre-votes, self-grant in raft_state_t */
|
||||
int raft_on_pre_vote_request(raft_state_t *s,
|
||||
int from_idx, uint64_t cand_term); /* 1=grant, 0=deny */
|
||||
int raft_on_pre_vote_reply(raft_state_t *s,
|
||||
int peer_idx, int granted); /* tally in raft_state_t -> WON / LOST / PENDING */
|
||||
```
|
||||
- Preconditions: `raft_begin_pre_vote` requires `role == RAFT_FOLLOWER`.
|
||||
- Postconditions: on `RAFT_PRE_VOTE_WON` the caller invokes the unchanged `raft_start_election`.
|
||||
- Invariants: persistent state untouched (asserted in `test_raft.c`).
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: `on_vote_eligible` is the existing `raft_cb_vote_eligible`; reusing it gives PRIMARY-deny + heartbeat-stickiness + catch-up-deny for free (Req 3.x).
|
||||
- Validation: unit tests assert `term`/`voted_for` unchanged after a granted PreVote.
|
||||
- Risks: accidental term mutation — mitigated by keeping all term writes in `raft_start_election` only.
|
||||
|
||||
### Controller — PreVote dispatch (cluster.c)
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Answer `preVoteRequest` with freshness + stickiness gates, no mutation; route `preVoteReply` to the tally |
|
||||
| Requirements | 2.1, 2.2, 3.1–3.3, 7.1, 7.2 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `OPCode_preVoteRequest`: parse `request_vote_req_t`; compute freshness floor via `ha_vote_self_freshness`; deny on freshness failure; otherwise call `raft_on_pre_vote_request`; reply `request_vote_reply_t` with **unchanged** `term`, the grant bit, and `peer_config_tag`/`peer_last_log_lsn`. Log denials with candidate id + reason (Req 7.2).
|
||||
- `OPCode_preVoteReply`: route to `pre_vote_reply_dispatch` → `raft_on_pre_vote_reply` (raft-library tally), paralleling the production `vote_reply_dispatch` → `raft_on_vote_reply`.
|
||||
- Inbound connections remain vote-only; the reply carries no state push.
|
||||
|
||||
**Contracts**: Service [x] / Event [x]
|
||||
|
||||
##### Event Contract
|
||||
- Published: `preVoteRequest` (term+1, candidate_id, config_tag, last_log_lsn).
|
||||
- Subscribed: `preVoteReply` (term, vote_granted, peer_config_tag, peer_last_log_lsn).
|
||||
- Delivery: best-effort over the existing election transport; missing/denied replies count as non-grants. No ordering requirement beyond per-peer idempotency: `raft_on_pre_vote_reply` receives `peer_idx` and tracks a per-peer replied flag in `raft_state_t` to avoid double-counting duplicate replies.
|
||||
|
||||
### Controller — PreVote orchestration & trigger restructuring (ha.c)
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Run the side-effect-free round, gate the failover trigger, collect replies, branch to real election or clean abandon |
|
||||
| Requirements | 1.1, 1.4, 1.5, 4.1–4.4, 5.4, 7.3 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `ha_begin_failover`: single entry from both `ha_on_primary_ctrl_fd_rdhup` and `liveness_poll_fn`; starts `ha_start_prevote_async`; **does not** truncate WAL or transition to `CTRL_ELECTING` (deferred to PreVote success).
|
||||
- `ha_start_prevote_async`: call `raft_begin_pre_vote` (self-grant + send `preVoteRequest` per peer via `on_send_pre_vote_request`); arm an `election_timeout_ms` round timeout. The PreVote tally lives in `raft_state_t`, not a controller struct.
|
||||
- `pre_vote_reply_dispatch` → `raft_on_pre_vote_reply`: on `RAFT_PRE_VOTE_WON` → truncate + `CTRL_ELECTING` + `ha_start_election_async`; on `RAFT_PRE_VOTE_LOST` or round timeout → stay `CTRL_STANDBY`, re-arm liveness, log abandonment (Req 7.3).
|
||||
- `ha_on_heartbeat_received`: if a PreVote round is in progress, cancel it (clear the round handle) and re-attach as standby (Req 4.2) — no term change.
|
||||
- **Invariant**: while a PreVote round is in progress, `ha->role` stays `CTRL_STANDBY` and no WAL/term mutation occurs.
|
||||
|
||||
**Contracts**: Service [x] / State [x]
|
||||
|
||||
##### State Management
|
||||
- State model: the PreVote tally (`pre_votes_granted`, `pre_vote_replies`, `pre_vote_in_progress`) lives in `raft_state_t`, mirroring the real-vote tally; quorum formula `(n_peers / 2) + 1` reused from `raft_on_vote_reply`. The controller holds only a round-timeout handle + in-progress flag for cancellation.
|
||||
- Concurrency: PreVote reply handling runs on the same single-threaded dispatch path as `vote_reply_dispatch`; no new lock introduced.
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: PreVote success calls the **unchanged** `ha_start_election_async`/`raft_start_election`, so the real election (and bug-339/340 behavior) is unaffected.
|
||||
- Validation: `test_ha_prevote.c` asserts the frozen-rejoin no-op (term constant, primary not demoted) and genuine-failover success.
|
||||
- Risks: a failed PreVote leaving the watchdog un-armed (would freeze failover) — covered by an explicit re-arm test.
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Strategy
|
||||
- **No PreVote majority** (expected, frozen-rejoin or split): not an error — abandon cleanly, stay `CTRL_STANDBY`, re-arm liveness, log at debug/info with reason. The cluster term is unchanged.
|
||||
- **Round timeout**: treated identically to "lost"; the next liveness cycle retries PreVote, succeeding once a majority's stickiness lapses (bounds legitimate failover by heartbeat-timeout, not unbounded — Req 5.4).
|
||||
- **Reply from a higher-term peer**: the PreVote reply path does **not** adopt the peer's term (PreVote is side-effect-free); a genuinely higher term is observed and adopted only through the normal real-vote / heartbeat paths, unchanged.
|
||||
- **Unknown candidate id / malformed payload**: deny PreVote (same as real-vote dispatch), log.
|
||||
|
||||
### Monitoring
|
||||
- Log every PreVote denial with candidate id + reason (`primary`, `fresh_heartbeat`, `catch_up`, `stale_term`, `freshness`) — Req 7.2.
|
||||
- Log campaign abandonment distinctly from genuine failover so operators can tell a suppressed disruptive election from a real one — Req 7.3.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (raft library — `test_raft.c`)
|
||||
- `raft_on_pre_vote_request` grants for a fresh, higher-term candidate when `on_vote_eligible` returns 1, and asserts `term`/`voted_for`/`role` unchanged (2.3, 6.2).
|
||||
- `raft_on_pre_vote_request` denies on `cand_term < term` and when `on_vote_eligible` returns 0 (2.2, 3.x).
|
||||
- `raft_on_pre_vote_reply` returns WON at quorum, LOST when all replied without quorum, with no role/term change (1.4, 1.5, 6.2).
|
||||
- A granted PreVote does not set `voted_for`, so a later real vote in the same term is still grantable (2.4).
|
||||
|
||||
### Integration Tests (controller — `test_ha_prevote.c`, `test_ha_vote_gate.c`)
|
||||
- **Frozen-then-resumed no-op**: 3-node, primary alive; a resumed standby starts PreVote; both peers deny (PRIMARY + fresh heartbeat); assert no real RequestVote sent, cluster term unchanged, primary still `CTRL_PRIMARY` (4.1, 4.3, 4.4).
|
||||
- **Genuine failover**: primary dead, peers' heartbeats stale; PreVote granted → real election → new primary within bounds (5.1, 5.4).
|
||||
- **Re-attach on heartbeat mid-round**: PreVote in progress, primary heartbeat arrives → round cancelled, node re-attaches as standby, no term change (4.2).
|
||||
- **PreVote eligibility reuse**: PRIMARY-deny, fresh-heartbeat-deny, catch-up-deny exercised through the PreVote path (3.1–3.3).
|
||||
- **Liveness re-arm after loss**: failed PreVote re-arms the watchdog; a subsequent genuine failure still triggers (1.4 robustness).
|
||||
|
||||
### Verification (Req 6)
|
||||
- `test_raft.c` invariant assertions and a `raft/raft.tla` consistency argument (RN-1, tracked in `research.md`) confirm Election Safety and term monotonicity hold with the PreVote path; the real-vote path is unchanged so bug-339/340 coverage is preserved (5.3).
|
||||
211
.kiro/specs/ha-prevote/requirements.md
Normal file
211
.kiro/specs/ha-prevote/requirements.md
Normal file
|
|
@ -0,0 +1,211 @@
|
|||
# Requirements Document
|
||||
|
||||
## Project Description (Input)
|
||||
|
||||
**Who has the problem.** Operators of XCE 3-node controller HA clusters, and the
|
||||
controllers themselves, when a controller is temporarily unreachable to its
|
||||
peers but not crashed — e.g. a process freeze (SIGSTOP, long GC/allocator
|
||||
stall), a one-way or transient network partition, or a slow host — and then
|
||||
becomes reachable again ("frozen-then-resumed" / "rejoining" node).
|
||||
|
||||
**Current situation.** The XCE election layer (Raft library `src/lib/raft/raft.c`
|
||||
plus the controller adapter in `src/controller/cluster.c`, `ha.c`, `myapp.c`)
|
||||
has no PreVote phase. A node that misses heartbeats hits its liveness timeout
|
||||
(`liveness_poll_fn` / RDHUP → `ha_on_primary_ctrl_fd_rdhup`) and immediately
|
||||
starts a real election, which **increments its persistent term** before any RPC
|
||||
(`raft_start_election`, `raft.c:68`). When such a node rejoins with an inflated
|
||||
term and sends `RequestVote`, the live primary adopts the higher term and steps
|
||||
down: `raft_on_vote_request` (`raft.c:125-134`, Step 2) updates `term`, sets
|
||||
role to follower, and fires `on_become_follower` **before** any freshness or
|
||||
role gate. The controller dispatcher (`cluster.c` `OPCode_requestVote`) only
|
||||
guards this with the `freshness_ok` check, so a rejoining node that was
|
||||
caught-up before the stall is fresh enough to pass and topples the leader.
|
||||
Because `raft_cb_become_follower` treats a was-PRIMARY demotion as
|
||||
demotion-convergence (truncate-above-committed-LSN + **self-restart**,
|
||||
`ha.c`), a single disruptive `RequestVote` makes the healthy primary truncate
|
||||
and restart, triggering a multi-term re-election storm.
|
||||
|
||||
Observed live (2026-06-15, 3-node ASAN deploy): a SIGSTOP'd-then-resumed standby
|
||||
forced the primary to demote and self-restart; terms climbed 3 → 6 over ~30s
|
||||
before the cluster reconverged. No committed data was lost (`committed_lsn` and
|
||||
all submitted jobs survived), and the cluster self-healed — but availability
|
||||
blipped for ~30s on an event that should be a no-op. A normal crash+restart does
|
||||
NOT trigger this: a restarted node restores its term from disk and observes
|
||||
`leader already known → skipping own election` (`myapp.c`). The problem is
|
||||
specific to nodes that inflate their term without a genuine leadership change.
|
||||
|
||||
This is the classic Raft "disruptive server" problem (Raft dissertation §9.6).
|
||||
The repository's TLA+ reference (`raft/raft.tla`) and the controller CLAUDE.md
|
||||
verification checklist require term monotonicity and Election Safety to be
|
||||
preserved by any change to this path.
|
||||
|
||||
**What should change.** Introduce a PreVote phase so a node cannot inflate its
|
||||
term or disrupt a healthy leader merely by becoming temporarily unreachable.
|
||||
Before incrementing its term and campaigning, a candidate must first win a
|
||||
PreVote majority — a hypothetical "would you vote for me in term+1?" round that
|
||||
does NOT change any node's persistent term or recorded vote. Peers grant a
|
||||
PreVote only if they (a) would grant the corresponding real vote under the
|
||||
existing freshness/log-completeness gate AND (b) have not heard from a current
|
||||
leader within the minimum election timeout (leader stickiness). A node that has
|
||||
a live primary connection (fresh heartbeat / valid `primary_ctrl_fd`) denies
|
||||
PreVotes, so a rejoining node in a cluster with a healthy leader gets no PreVote
|
||||
majority, never increments its term, and simply re-attaches as a standby when
|
||||
the primary's next heartbeat/ctrlhello arrives.
|
||||
|
||||
The change must:
|
||||
- Preserve all existing Raft safety invariants (Election Safety ≤1 leader/term,
|
||||
Leader Completeness, term monotonicity, monotonic commit index) and remain
|
||||
consistent with `raft/raft.tla` and the controller verification checklist.
|
||||
- Not regress the already-fixed failover (bug-339) and rejoin (bug-340) paths,
|
||||
nor the legitimate cases where a real leadership change SHOULD occur (true
|
||||
primary death, network healing into a higher-term majority).
|
||||
- Integrate with the existing two-tier transport (capnp `RequestVote` /
|
||||
`RequestVoteReply` over iomgr, the `ha_elect_ctx_t` vote collection, and the
|
||||
raft library callback seams) without breaking the unit-test seams.
|
||||
|
||||
Scope: the raft library election state machine and its PreVote RPC, the
|
||||
controller vote dispatch/freshness/heartbeat-stickiness gates, the election
|
||||
trigger path (liveness timeout / RDHUP → PreVote → real election), and the
|
||||
wire message (a PreVote flag on the existing vote RPC or a new PreVote message).
|
||||
Out of scope: changes to replication, the commit engine, snapshot transfer, or
|
||||
config bootstrap beyond what the PreVote gating requires.
|
||||
|
||||
## Introduction
|
||||
|
||||
This feature adds a **PreVote phase** to the XCE controller HA election path so
|
||||
that a controller which is merely temporarily unreachable cannot inflate its
|
||||
term or unseat a healthy primary. Before a node increments its persistent term
|
||||
and starts a real election, it must first win a hypothetical PreVote majority.
|
||||
Peers grant a PreVote only when the candidate could legitimately become leader
|
||||
(the existing freshness / log-completeness gate) **and** the peer no longer
|
||||
believes a current leader is alive (leader stickiness). The net effect: a
|
||||
frozen-then-resumed standby in a cluster with a healthy primary gets no PreVote
|
||||
majority, never increments its term, and re-attaches as a standby — while
|
||||
genuine leadership changes (true primary death, healing into a higher-term
|
||||
majority) still proceed. The change must preserve every existing Raft safety
|
||||
invariant and must not regress the previously fixed failover (bug-339) and
|
||||
rejoin (bug-340) behavior.
|
||||
|
||||
Throughout this document, "the Election Module" refers to the controller's HA
|
||||
election state machine (raft library plus its controller adapter) as observed
|
||||
through cluster behavior; "the voting controller" refers to a peer evaluating an
|
||||
incoming PreVote request; and "the HA cluster" refers to the externally observed
|
||||
behavior of the 3-node controller cluster as a whole.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**:
|
||||
- A PreVote round that precedes any persistent term increment and real
|
||||
election on the election-trigger path (liveness timeout / primary-connection
|
||||
loss).
|
||||
- Voter-side PreVote evaluation combining the existing freshness /
|
||||
log-completeness gate with a new leader-stickiness gate.
|
||||
- The user-/operator-observable outcomes for frozen-then-resumed rejoin,
|
||||
genuine failover, and partition healing.
|
||||
- PreVote requests and replies traveling over the existing election transport.
|
||||
- **Out of scope**:
|
||||
- Replication, the commit/barrier engine, snapshot/WAL catch-up transfer, and
|
||||
config bootstrap — except where PreVote gating must read state those paths
|
||||
already expose.
|
||||
- Redesign of the existing freshness / log-completeness comparison itself; this
|
||||
feature reuses it as the real-vote eligibility criterion.
|
||||
- **Adjacent expectations** (relied on, not owned by this feature):
|
||||
- The existing freshness / log-completeness gate (config-tag and last-log-LSN
|
||||
floor) remains the authority on whether a candidate could win a real vote.
|
||||
- The existing heartbeat-liveness / primary-connection tracking remains the
|
||||
signal for whether a peer still believes a leader is alive.
|
||||
- The existing two-tier election transport and vote-collection mechanism
|
||||
continue to deliver and tally PreVote and real-vote messages.
|
||||
- Term restore-from-disk on crash+restart continues to suppress unnecessary
|
||||
elections; this feature does not change that path.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: PreVote precedes any persistent term increment
|
||||
|
||||
**Objective:** As an XCE controller node, I want to win a hypothetical PreVote
|
||||
majority before starting a real election, so that I never inflate my persistent
|
||||
term unless a genuine leadership change is actually possible.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a node's election trigger fires (liveness/heartbeat timeout or primary-connection RDHUP), the Election Module shall initiate a PreVote round before incrementing its persistent term.
|
||||
2. While a PreVote round is in progress, the Election Module shall not modify the node's persistent term or recorded vote.
|
||||
3. When a node requests PreVotes, the Election Module shall request them for the prospective next term (current term + 1) without persisting that prospective term.
|
||||
4. If a node does not obtain a PreVote majority, then the Election Module shall not start a real election and shall not increment its persistent term.
|
||||
5. When a node obtains a PreVote majority, the Election Module shall start a real election by incrementing its persistent term and casting its self-vote, following the existing real-election path.
|
||||
|
||||
### Requirement 2: Voter-side PreVote eligibility reuses the real-vote gate
|
||||
|
||||
**Objective:** As a voting controller, I want to grant a PreVote only to a
|
||||
candidate I would actually elect, so that PreVote screens out candidates that
|
||||
could never legitimately become leader.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a voting controller receives a PreVote request, the voting controller shall grant it only if the candidate would also pass the existing real-vote freshness and log-completeness gate (config-tag and last-log-LSN floor).
|
||||
2. If a PreVote candidate fails the freshness or log-completeness gate, then the voting controller shall deny the PreVote.
|
||||
3. When a voting controller grants or denies a PreVote, the voting controller shall not record a vote, shall not change its voted-for state, and shall not change its persistent term.
|
||||
4. When a voting controller has granted a PreVote to one candidate, the voting controller shall remain able to grant a real vote later in the same term to any eligible candidate, regardless of which candidate received the PreVote.
|
||||
|
||||
### Requirement 3: Leader stickiness gate
|
||||
|
||||
**Objective:** As a voting controller that still believes its leader is alive, I
|
||||
want to deny PreVotes, so that a temporarily-unreachable peer cannot disrupt a
|
||||
healthy primary.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. While a voting controller still considers a current leader alive — it has received leader traffic recently enough that it would not itself start an election — the voting controller shall deny all PreVote requests, even when the candidate passes the freshness gate.
|
||||
2. If a voting controller has gone long enough without leader traffic that it would itself be eligible to start an election, then the voting controller shall evaluate PreVote requests under the freshness / log-completeness gate alone.
|
||||
3. While a controller is itself the current primary holding quorum, the controller shall deny all PreVote requests.
|
||||
|
||||
### Requirement 4: Frozen-then-resumed rejoin is non-disruptive
|
||||
|
||||
**Objective:** As an operator, I want a frozen-then-resumed standby to rejoin
|
||||
without disrupting the cluster, so that transient unreachability is effectively a
|
||||
no-op for availability.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When a previously-frozen standby resumes and initiates a PreVote round while a healthy primary still holds the cluster, the HA cluster shall deny it a PreVote majority so that it neither increments its term nor sends a real RequestVote.
|
||||
2. When a non-disruptive rejoining standby next receives a primary heartbeat or ctrlhello, the standby shall re-attach as a standby under the existing primary without any intervening persistent-term change.
|
||||
3. While a healthy primary holds quorum, the HA cluster shall not demote, truncate-and-converge, or self-restart the primary in response to a rejoining standby's election attempt.
|
||||
4. When a frozen-then-resumed event occurs in an otherwise healthy 3-node cluster, the HA cluster shall keep the cluster's persistent term unchanged across the event and shall not enter a multi-term re-election cycle.
|
||||
|
||||
### Requirement 5: Legitimate leadership changes still proceed
|
||||
|
||||
**Objective:** As an XCE HA cluster, I want genuine leadership changes to
|
||||
complete, so that PreVote suppresses only disruptive elections and never blocks
|
||||
real failover.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the primary genuinely fails and no peer has heard from a leader within its election-eligibility window, the HA cluster shall grant PreVotes to an eligible candidate so that a real election proceeds and a new primary is elected.
|
||||
2. When a partition heals and a node observes a higher-term majority, the Election Module shall still adopt the higher term and step down according to the existing term-monotonicity rules.
|
||||
3. The Election Module shall preserve the previously fixed failover behavior (bug-339) and rejoin behavior (bug-340) with PreVote enabled.
|
||||
4. Where a real leadership change is warranted, the HA cluster shall complete the new election within the existing election-timeout bounds, introducing no additional unbounded delay from the PreVote round.
|
||||
|
||||
### Requirement 6: Raft safety invariants preserved
|
||||
|
||||
**Objective:** As an XCE HA cluster, I want every existing Raft safety invariant
|
||||
preserved, so that adding PreVote introduces no correctness regression.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Election Module shall preserve Election Safety (at most one leader per term) across all PreVote and real-vote paths.
|
||||
2. The Election Module shall preserve term monotonicity, such that no PreVote causes any node's persistent term to decrease and no node's persistent term increases without a completed real election.
|
||||
3. The HA cluster shall preserve Leader Completeness and a monotonic commit index across PreVote-gated elections.
|
||||
4. The PreVote behavior shall remain consistent with the repository's TLA+ reference (`raft/raft.tla`) and the controller verification checklist for term monotonicity and Election Safety.
|
||||
|
||||
### Requirement 7: PreVote transport compatibility and operator observability
|
||||
|
||||
**Objective:** As an operator, I want PreVote activity to be diagnosable and to
|
||||
not break existing election exchanges, so that I can distinguish a suppressed
|
||||
disruptive election from a real failover.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The HA cluster shall carry PreVote requests and replies over the existing election transport without disrupting real RequestVote / RequestVoteReply exchanges.
|
||||
2. When a controller denies a PreVote due to leader stickiness or the freshness gate, the HA cluster shall record the denial with the candidate identity and the denial reason at a diagnosable log level.
|
||||
3. When a node abandons a campaign after failing to obtain a PreVote majority, the HA cluster shall record that outcome so operators can distinguish a suppressed disruptive election from a genuine failover.
|
||||
22
.kiro/specs/ha-prevote/spec.json
Normal file
22
.kiro/specs/ha-prevote/spec.json
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"feature_name": "ha-prevote",
|
||||
"created_at": "2026-06-15T23:45:43Z",
|
||||
"updated_at": "2026-06-16T00:55:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true
|
||||
}
|
||||
|
|
@ -21,7 +21,7 @@ XCE is an HPC (High-Performance Computing) cluster management system providing j
|
|||
|
||||
XCE provides a **fully integrated single-vendor stack** (no database setup, unified build system) with an open, buildable codebase (Nix flakes, AppImage packaging). Compared to Slurm/PBS, it eliminates the external database requirement. Compared to commercial LSF, it is open source and reproducibly buildable.
|
||||
|
||||
**Key architectural trade-off**: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via `raft.c`/`raft.h`, and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, `committed_lsn` ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
|
||||
**Key architectural trade-off**: single-writer model throughout — only the primary writes. HA follows Raft semantics with CP guarantees: consensus election (RequestVote, term-based voting, freshness gates) via the `src/lib/raft/` shared module (`raft.c`/`raft.h`, consumed by the controller through `ha_init_raft`), and majority-ACK log commitment for state-changing writes (WAL entries ≈ Raft log, `committed_lsn` ≈ commitIndex). Operations that cannot reach majority are rejected, never locally degraded. The single-writer constraint is XCE's simplification of Raft — no concurrent client proposals — while commitment and election inherit Raft's safety rules. All controller state is in-memory; persistence is WAL + checkpoint, not a traditional database.
|
||||
|
||||
## Active Development Context
|
||||
|
||||
|
|
@ -39,10 +39,13 @@ The project is in a **persistence and HA transformation** (branch `persistent_ha
|
|||
|
||||
Features are designed and implemented via the kiro spec system (`.kiro/specs/`).
|
||||
Each feature follows a 3-phase approval workflow: Requirements → Design → Tasks → Implementation.
|
||||
Active specs (10 total) include: ha-config-bootstrap, ha-heartbeat-liveness,
|
||||
Active specs (15 total). The HA consistency family is the current focus:
|
||||
ha-consistency-invariants (umbrella contract) decomposed into ha-commit-engine,
|
||||
ha-commit-barrier-rollout, ha-single-leader-role, and ha-convergence-recovery.
|
||||
Supporting/foundational specs: ha-config-bootstrap, ha-heartbeat-liveness,
|
||||
ha-peer-connection-reuse, ha-repl-hardening, ha-wal-replication, raft-library,
|
||||
election-iomgr-refactor, scheduler-controller-msg-gap, worker-join-refactor,
|
||||
resval-id-decoupling.
|
||||
resval-id-decoupling. See `roadmap.md` for dependency order and status.
|
||||
|
||||
Key design decisions already locked:
|
||||
- No `fork()` for snapshots (mutex inheritance deadlock in multithreaded controller)
|
||||
|
|
|
|||
|
|
@ -36,6 +36,8 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
|
|||
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
|
||||
- [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_
|
||||
- [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
|
||||
- [ ] xce_test shell command support — add `!<command>` prefix to `xce_test` test scripts to execute arbitrary shell commands inline during test runs. Enables test scripts to drive environment setup/teardown (e.g., kill a node, write a file, run a helper) without calling out to a separate harness layer. Scope: parser change in the xce_test runner to detect `!`-prefixed lines, fork/exec the remainder as a shell command, fail the test on non-zero exit, capture stdout/stderr to the test log.
|
||||
- [ ] scheduler framework enhancement — strengthen the scheduler subsystem: (a) improve job scheduling policy extensibility (pluggable queue disciplines beyond FIFO), (b) add priority-aware dispatch, (c) improve resource-aware scheduling (CPU/memory/GPU slot tracking per host), (d) wire scheduler decisions through the commit barrier so scheduling state is replicated and survives failover. Dependencies: ha-consistency-invariants (commit barrier). Spec advised before implementation; promote via `/kiro-spec-init "scheduler-framework"` when ready.
|
||||
|
||||
## Superseded / Absorbed Items
|
||||
|
||||
|
|
|
|||
|
|
@ -7,6 +7,8 @@
|
|||
```
|
||||
src/
|
||||
├── lib/ C — shared library (libxce): data structures, IPC, utilities
|
||||
│ ├── raft/ C — reusable Raft consensus module (raft.c/raft.h)
|
||||
│ └── testsuite/ C — libxce unit tests (test_*.c)
|
||||
├── controller/ C — cluster controller daemon
|
||||
├── worker/ C — per-node job execution daemon (xcecd)
|
||||
├── scheduler/ C — job scheduling engine (plugin architecture)
|
||||
|
|
|
|||
|
|
@ -97,4 +97,4 @@ Early return on every error. Log with zap fields before returning.
|
|||
3. **CGO bridge** — `backend/` package wraps C library calls; unit tests run `CGO_ENABLED=0`, backend tests need `CGO_ENABLED=1` inside nix dev shell.
|
||||
4. **Swagger-first API docs** — `swaggo/swag` annotations on handlers; docs served at runtime by Gin.
|
||||
5. **Structured logging** — every log entry carries context via `zap.String()`, `zap.Error()`. Never use `fmt.Println` for operational output.
|
||||
6. **Single-writer HA** — controller is always one active primary; standby election via Raft-style consensus (term-based voting, freshness gates) in `raft.c`, with the single-writer constraint avoiding Raft log replication complexity.
|
||||
6. **Single-writer HA** — controller is always one active primary; standby election via Raft-style consensus (term-based voting, freshness gates) in the `src/lib/raft/` shared module (`raft.c`/`raft.h`), with the single-writer constraint avoiding Raft log replication complexity.
|
||||
|
|
|
|||
353
.wolf/anatomy.md
353
.wolf/anatomy.md
|
|
@ -1,7 +1,7 @@
|
|||
# anatomy.md
|
||||
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-06-12T15:29:40.114Z
|
||||
> Files: 515 tracked | Anatomy hits: 0 | Misses: 0
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-06-16T07:17:57.828Z
|
||||
> Files: 507 tracked | Anatomy hits: 0 | Misses: 0
|
||||
|
||||
## ./
|
||||
|
||||
|
|
@ -36,6 +36,10 @@
|
|||
|
||||
- `SKILL.md` — kiro-impl Skill (~4048 tok)
|
||||
|
||||
## .claude/skills/kiro-impl/.claude/
|
||||
|
||||
- `ucai-formatter-cache.local.json` (~17 tok)
|
||||
|
||||
## .claude/skills/kiro-impl/templates/
|
||||
|
||||
- `debugger-prompt.md` — Debug Investigator (~812 tok)
|
||||
|
|
@ -177,7 +181,7 @@
|
|||
|
||||
## .claude/skills/xce-deploy-ha/
|
||||
|
||||
- `SKILL.md` — XCE HA Cluster Deployment (~1334 tok)
|
||||
- `SKILL.md` — XCE HA Cluster Deployment (~1345 tok)
|
||||
|
||||
## .code-review-graph/
|
||||
|
||||
|
|
@ -186,8 +190,8 @@
|
|||
## .codegraph/
|
||||
|
||||
- `.gitignore` — Git ignore rules (~47 tok)
|
||||
- `codegraph.db-shm` (~8738 tok)
|
||||
- `daemon.log` (~3580 tok)
|
||||
- `codegraph.db-shm` (~8739 tok)
|
||||
- `daemon.log` (~9499 tok)
|
||||
- `daemon.pid` (~44 tok)
|
||||
|
||||
## .kiro/settings/rules/
|
||||
|
|
@ -252,8 +256,8 @@
|
|||
- `design.md` — Design: ha-commit-barrier-rollout (~10372 tok)
|
||||
- `requirements.md` — Requirements Document (~3181 tok)
|
||||
- `research.md` — Research & Design Decisions: ha-commit-barrier-rollout (~3672 tok)
|
||||
- `spec.json` (~143 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-commit-barrier-rollout (~4663 tok)
|
||||
- `spec.json` (~341 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-commit-barrier-rollout (~5658 tok)
|
||||
|
||||
## .kiro/specs/ha-commit-engine/
|
||||
|
||||
|
|
@ -290,7 +294,7 @@
|
|||
- `requirements.md` — Requirements Document (~3586 tok)
|
||||
- `research.md` — Research & Design Decisions: ha-convergence-recovery (~3542 tok)
|
||||
- `spec.json` (~142 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-convergence-recovery (~3434 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-convergence-recovery (~5679 tok)
|
||||
|
||||
## .kiro/specs/ha-heartbeat-liveness/
|
||||
|
||||
|
|
@ -309,6 +313,14 @@
|
|||
- `spec.json` (~169 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-peer-connection-reuse (~2022 tok)
|
||||
|
||||
## .kiro/specs/ha-prevote/
|
||||
|
||||
- `design.md` — Design Document (~6905 tok)
|
||||
- `requirements.md` — Requirements Document (~3604 tok)
|
||||
- `research.md` — Gap Analysis: ha-prevote (~6077 tok)
|
||||
- `spec.json` (~126 tok)
|
||||
- `tasks.md` — Implementation Plan (~3129 tok)
|
||||
|
||||
## .kiro/specs/ha-repl-hardening/
|
||||
|
||||
- `design.md` — Design: HA Replication Hardening (~5577 tok)
|
||||
|
|
@ -325,7 +337,7 @@
|
|||
- `requirements.md` — Requirements Document (~2177 tok)
|
||||
- `research.md` — Research & Discovery Log: ha-single-leader-role (~3970 tok)
|
||||
- `spec.json` (~142 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-single-leader-role (~3232 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-single-leader-role (~5040 tok)
|
||||
|
||||
## .kiro/specs/ha-wal-replication/
|
||||
|
||||
|
|
@ -372,12 +384,12 @@
|
|||
- `api-standards.md` — API Standards (~1291 tok)
|
||||
- `build-deployment.md` — Build & Deployment (~1241 tok)
|
||||
- `error-handling.md` — Error Handling Standards (~782 tok)
|
||||
- `ha-async-io.md` — HA Protocol & Async I/O (~1360 tok)
|
||||
- `product.md` — Product Overview (~1155 tok)
|
||||
- `roadmap.md` — Roadmap (~2206 tok)
|
||||
- `ha-async-io.md` — HA Protocol & Async I/O (~1431 tok)
|
||||
- `product.md` — Product Overview (~1246 tok)
|
||||
- `roadmap.md` — Roadmap (~2623 tok)
|
||||
- `security.md` — Security Standards (~804 tok)
|
||||
- `structure.md` — Project Structure (~1139 tok)
|
||||
- `tech.md` — Technology Stack (~1144 tok)
|
||||
- `structure.md` — Project Structure (~1170 tok)
|
||||
- `tech.md` — Technology Stack (~1155 tok)
|
||||
- `testing.md` — Testing Standards (~741 tok)
|
||||
|
||||
## docs/superpowers/plans/
|
||||
|
|
@ -393,328 +405,311 @@
|
|||
|
||||
## graphify-out/
|
||||
|
||||
- `.graphify_labels.json` (~5762 tok)
|
||||
- `.graphify_labels.json` (~6030 tok)
|
||||
- `.graphify_root` (~1 tok)
|
||||
- `GRAPH_REPORT.md` — Graph Report - persistent_ha (2026-06-12) (~46969 tok)
|
||||
- `manifest.json` (~35204 tok)
|
||||
- `GRAPH_REPORT.md` — Graph Report - persistent_ha (2026-06-16) (~49152 tok)
|
||||
- `manifest.json` (~35956 tok)
|
||||
|
||||
## graphify-out/cache/
|
||||
|
||||
- `stat-index.json` (~44692 tok)
|
||||
- `stat-index.json` (~45861 tok)
|
||||
|
||||
## graphify-out/cache/ast/
|
||||
|
||||
- `000c131d43cb1e40a783245cb81906a2abd9297491964242e5b5bbebcad1b0d1.json` (~10944 tok)
|
||||
- `004cc8d07f2f670da29d7c7f144c8875f274b8438fb1d4306fb6cc676c745fea.json` (~1050 tok)
|
||||
- `005d423b344fb7001a76e9a01e7226b28957f0318f503b6acd0f7f72a3fcde8e.json` (~227 tok)
|
||||
- `008610a0f4315b9a68f7cde989a73b27de885c7ab2c96a9e7440be16cf2fa094.json` (~2422 tok)
|
||||
- `00a44f36a2da752dce1be0700e58c3c4c485f8fd75e462e12d15f4ba144d9bf2.json` (~29528 tok)
|
||||
- `00a5ff6db9d63a87d19ca8b0a68d7105c6ea2fa89be5f94c98b257bced832042.json` (~11282 tok)
|
||||
- `00bf4ac62988ea0494c2ffda33b92affa91df5763081a54b4c7d6c308fec999e.json` (~269 tok)
|
||||
- `010e4ba8e25fd6fecc0ab19203ed3f62aeb3bb1a42aa4b4712f2040eb7816392.json` (~10023 tok)
|
||||
- `01181572b4bf0d966f8391007f85c7a58838d83ef521a9fb455e180c704c78be.json` (~4182 tok)
|
||||
- `012463e815ba3f31b95b398fa0242f2f331578618542a0910dfc9e9be8c2a0a1.json` (~15340 tok)
|
||||
- `0158107a72d3c8414a7500cf543421f458e5cfff4a912448a50e1e37aae35b9a.json` (~21697 tok)
|
||||
- `016845199833895f368a40f072564f7b5502cd652cef19f58dffb9c1931bd179.json` (~1092 tok)
|
||||
- `019ce52f853a0d7c2d8646054bab194a2217f217ddf3d0ed4d0d476ec4553329.json` (~8110 tok)
|
||||
- `01c91aa6fa18cc3129d3f6451a04b235c9528a88a165b1b6126f87c37bdc4048.json` (~14877 tok)
|
||||
- `01d5ae01a90dcbf2812ac05362927a7f9320cd5f20875838057caf559be8c5be.json` (~224 tok)
|
||||
- `01d75ef42f259e5752e9ee274fc1e00781b485b87349841c5c335ad92ba964b9.json` (~6308 tok)
|
||||
- `01ffefd3077e3989045ab89dbd8d9951d161531a0bef7cee8f652de34593ef79.json` (~40640 tok)
|
||||
- `0211c966a13950b059d40d37ff94e62d5b7ba25b554407097e4f964eaac39102.json` (~206 tok)
|
||||
- `023e107588bcbc8780b5fe90b346a96c5243e9192778aefb9bc9fcd92b5cf692.json` (~3708 tok)
|
||||
- `027186d92c26c611e9c4d3a9463dde3d13a74de8936252d1f0086ef64f4c11d5.json` (~12735 tok)
|
||||
- `02792109a97363521613514c608dc94da901e12328ff8ed7925297498b49e28b.json` (~11713 tok)
|
||||
- `0297e5997050ad938795b7335ba2e08338aa3e4b36046f19b34c3069fd8d4e60.json` (~57184 tok)
|
||||
- `02b1338304d9a7a16ca3476f5c5f60ba7b9e1d6f8aeb10fad01a7d8be2fc7bad.json` (~14324 tok)
|
||||
- `0327a8119a2ad0c8ee5fda692968124551ae97ec306a8d102349b9f9f9c587aa.json` (~652 tok)
|
||||
- `0334c7cb345f94591499afbcd6d06808702619a27ad0cad749c9ccf84738f77c.json` (~8238 tok)
|
||||
- `037105385f82cad6a302e5f737557c04fe1ca516e5b0ec8b188550958df851cd.json` (~29997 tok)
|
||||
- `0401198757a993070c98c754996a960299d605d55ee524b2f1aad817fa94322a.json` (~3011 tok)
|
||||
- `0438e6d96c89517e2640f19067b6685b2fb95af119ecc5df3f896adbba6c1933.json` (~8120 tok)
|
||||
- `040ce3f94fd026464306f32ccbc5e0b59881a0f5654574d0c8564f9d06143c72.json` (~23952 tok)
|
||||
- `0440846ef240f3abb98b1ca27304c5cdbe56779ba1cc857850dd7a26cad49b66.json` (~2935 tok)
|
||||
- `04583d91aa48af789b952136fb7c52041b0d632793246105f3af6fe070727a07.json` (~2049 tok)
|
||||
- `0490c7b9fd5bba4286f161b03204a38f9ec48a4889ab597b70a20172aacbf4c6.json` (~8120 tok)
|
||||
- `04e3a10a165fc6dea955a31ec5c48db1e75c23545a74a734cb538d8ef8ac06ef.json` (~7004 tok)
|
||||
- `053929f384ca0a4fdabda00415d3f0e0a7f8ccbd86f273deadd54ac0590492a2.json` (~18832 tok)
|
||||
- `0550909cf1a1da30dee6c2653f79809399374ee35919a467be9583fa7d27f5c5.json` (~71978 tok)
|
||||
- `053f96f9072f59c2f44a8390d4047be1fefb697ea6db29f1eeca60602b7ab759.json` (~396 tok)
|
||||
- `057d67f05183f6fe7fd3414cb6f46de234ebe58fa3dbca7dddc5441ee3f8a2f1.json` (~6218 tok)
|
||||
- `061bfdbdc03ae56a916d366f3200ca2c610d9ad141e11682e47243197d93632b.json` (~592 tok)
|
||||
- `06578221cb7a75ee06e0460d5a42c48037e3ddcebbc21fae45825886ceaa0355.json` (~15340 tok)
|
||||
- `0584ab00ab563c9f2f68e27d7bd9b58c07e95ff4c9ea2403a82818009186147f.json` (~81 tok)
|
||||
- `058ef9fb07d4ca45e2f49b6fa79609a1ce85c8662aa43696392b35bd9047e505.json` (~8010 tok)
|
||||
- `05b255f2ba83087a09d40fcf9e1f10209c1276b1e86c65d0eecdab2de2b28076.json` (~15467 tok)
|
||||
- `06a0b286e4c22fb4c981239820aea2042950d1d21dc96de40226b14cc696065e.json` (~8244 tok)
|
||||
- `06b99babef4c7ab862aa487145fb2e6736245ef4ad565fe0e06a24898c3875d7.json` (~352 tok)
|
||||
- `07014154947b85f86499894c59be83a316bf3fc006c8fc55c316cb893462977e.json` (~3083 tok)
|
||||
- `074a8bce08af84573c3e948b6dabbecdb6a7e99daca393ddaae4f816b1b24411.json` (~309 tok)
|
||||
- `074b4a288109008bdb35dccbcbc81a7e2b029bd17e4707fb967a94d962a77b69.json` (~4508 tok)
|
||||
- `0752d0d43dbdaac573d4c7129d228ad267a5a5d8bc6820fde4b6e6a6a3062100.json` (~586 tok)
|
||||
- `0799df8da46779b19ce58b3ae776614f647d6d5f009f418520eec8897f538f9d.json` (~1961 tok)
|
||||
- `07d0b1dbd63fc8b71cab7d602d56453c4ed0a4170b3efe8d387de29c09487ead.json` (~467 tok)
|
||||
- `07e3abfd90acf1c5d2611d59e5b4f199a6575888c2fb5e9e3b799ec4059c85e5.json` (~374 tok)
|
||||
- `07d9e114e326083e00a16d55832ae1225b103e3a506da26b69f58bee6a0f14c8.json` (~76 tok)
|
||||
- `082d2508e015e3002b8399b37a4562991cff2b2d747f1e800d370c1f3e88b028.json` (~5332 tok)
|
||||
- `0857f9be25466f6f3c0da48438f7c7cf35d0140010f4639a673c2f35ae98a002.json` (~2468 tok)
|
||||
- `0894c6309b9aae7fdb3b1aef7fb1dfaf1b55fd8f31ac96a8112916d55ebbedba.json` (~315 tok)
|
||||
- `0895960bce4ca499c4505de3f178508e1cd9dda69a3e8ab43195493957793a34.json` (~7103 tok)
|
||||
- `08e2728caebdaff0489f8e7bac962b9af1da74bab7072aa9b430024055889dfe.json` (~4802 tok)
|
||||
- `0933c8c03485fee87e59ee544f2baed215459e37d445226aba3990bb5460dd8b.json` (~14400 tok)
|
||||
- `09cb38261b447a3240b1361064d5bd8dada4c7eb8430b961c0476e902bc11ea0.json` (~8966 tok)
|
||||
- `09fc4c36a264a7b935bf7ffe9e8506fc3452b8e90c8ecb7e22d6a740854ebc76.json` (~1802 tok)
|
||||
- `0ac5f46d303d3f4b45966029f5cf09643eca5e14117e4976e097b8e83120401c.json` (~76 tok)
|
||||
- `0a65fc39bdd68ec3137a208b663c8827bf663aa6ba3cbcbb94ae03d0d3ba4fa2.json` (~81 tok)
|
||||
- `0b84abe002c91c4608ec56b28a1a39a066c1d64f8bad5182c89a905e0f51138f.json` (~6400 tok)
|
||||
- `0b91b1fcfc8ae3bd247107dd81b84a03aea885632599033c06e8f9f372f8d967.json` (~504 tok)
|
||||
- `0bcad3d5790defe9b965d14f4fdf8b5bb80273549a9872226fce33ee2453cfd6.json` (~1122 tok)
|
||||
- `0bf7323dd6f181b3c8536da6fda90904a6d3ea5fb99d9a9211175beadd33ae49.json` (~1973 tok)
|
||||
- `0bfddfa19374691c60a82a0208b83881a8c2c8adc9e888576e7aa64801be9f9b.json` (~4672 tok)
|
||||
- `0c349cee86e9ad96945f644460c1e3faa3c9af3df65079ae786207179039cd55.json` (~2248 tok)
|
||||
- `0c6b592cd952a83286c28fa23af1707fa0ec26c2c751337e7fe61c5f25179808.json` (~25338 tok)
|
||||
- `0c86267402031c08bcc3b4b5a1202712295c728654b1958f36368c636c0416a0.json` (~3581 tok)
|
||||
- `0ccb2e06c2139ef03aef6ba894efe1025c61daed9a9ef15b891319ecc4d9652e.json` (~5859 tok)
|
||||
- `0d10839a95bf89b63b21c49624d5447044dcb1e9264157ccde87e557edb0a640.json` (~28018 tok)
|
||||
- `0d31fe94cc46afc9674019867b10d786225c8a2af812798213e5aa9a960cc4f2.json` (~81 tok)
|
||||
- `0d5480993d38e8a128d23ff34faa9270999475e305b2f242bc82e4f674262739.json` (~15294 tok)
|
||||
- `0d75597651d23293ccc9bbe768fb9c1aeb53a0bd81b80a377996f5e68ad79fe6.json` (~1122 tok)
|
||||
- `0dab5b24f3cee60a70662d6ce8aef5a2bb4465361ba368b61f2d1c064c8a6cf1.json` (~269 tok)
|
||||
- `0db57ca4ac0c3ee6abc97242d0a04f274b18b83ba32d3b9c2b8c39b949c52fba.json` (~2878 tok)
|
||||
- `0dcea2652aaf077ffb9d6f61c387ac6dc42e03d347a5ea9dfde069c107ebabe4.json` (~26116 tok)
|
||||
- `0e688ceec91f4b7c192ec781324e9ddaa517ac653dc2256095ec17c49cb12928.json` (~309 tok)
|
||||
- `0eada2ef529a3511cb04b120cb3d2247c7822f9aa4c4d9dab4f7cfdc72d31e8a.json` (~1980 tok)
|
||||
- `0ebe1eba67e97d4287dcf82221f8fa6dceb6aef0980a011d21e7f7e8ce7ef741.json` (~592 tok)
|
||||
- `0ee129793c23e5933eade7198c1f08c5d84c9c26139e7eb7ff067d5c52d08d1e.json` (~6165 tok)
|
||||
- `0ef710579ddbf54eabe4f4ef870293e98d3ba4fc8f88351e31cbd7b579054d0b.json` (~60278 tok)
|
||||
- `0f1d7b66657a17d9d64f32e497db7dad40d2e9ddc16d9f96c9a789b750ec5cad.json` (~17210 tok)
|
||||
- `0f5341ff6ca93ba2938f8f8745a9739a2f5da1705bdb8b6abca03899d71e75c6.json` (~224 tok)
|
||||
- `0fa8ec51434b17edc5a26d6a037f6799618b873ebae92d6eac751f0f3578c038.json` (~2079 tok)
|
||||
- `0fb70f67355c6ee9bbb1dd9f5d55d21fdd019790f551898c33e6dd97d0742331.json` (~468 tok)
|
||||
- `0fd3a7a67fea0b2266942e702549a4de76ae86d31fbe66a68c21d765f66d37f0.json` (~15463 tok)
|
||||
- `10113aa2a94b02bb8464b4eaffd2ba059c6b9d386bd5f522eb4ec82d04a19bcb.json` (~17818 tok)
|
||||
- `103e27126a9e5950aa8c8063c54df342d6737d340c5411cf667b24a00dcc28a1.json` (~3251 tok)
|
||||
- `104ff5c6fbc4f31f0bc8dff467f2d17016e763fb171b757cc6eade51aab16220.json` (~10462 tok)
|
||||
- `105cfdcc702831a141523e3d24e7505c1da12134d0ddafee2000f43196596c54.json` (~7236 tok)
|
||||
- `111178800ae1876a431e527f3e0211c41e066afcf3636483674e1de0177fe726.json` (~3496 tok)
|
||||
- `1115d59cae0de03be0ebfeec800e4bc7a3142b1df07c22537ff00164f6d4becd.json` (~845 tok)
|
||||
- `114f295685c5796b4dcd914d8c84d0e74823a3000bce5dbf796f52519176b793.json` (~275 tok)
|
||||
- `11572830a363253a8645bfc2cb739462796329cfa74713b04ceab4b9abb7d6e9.json` (~60916 tok)
|
||||
- `1164b6d75143d4caf1a8c3ccd18258c793607dae4d869a5490fc09564b26c2af.json` (~214 tok)
|
||||
- `1196f6d0f42ffbd96922026767a312e1d25869970da006eda64602bb96691c22.json` (~228 tok)
|
||||
- `11a454492a215c5707704555427cc5a8a8d8c9e37140369fe3fd48964b6cd6be.json` (~3120 tok)
|
||||
- `11d0e37a4fec59ec78b60e8460549aaa4e31c61dda04508aae456719ef213d74.json` (~81 tok)
|
||||
- `12606235c64f192ab75e049b7dc8c659af9d9cefa4c8f0ecd043ddd1087ca8c6.json` (~81 tok)
|
||||
- `11a72a0ce21f2f1a221493c217301e9d2721e7c93ad8fbf3eace49ba689be674.json` (~15463 tok)
|
||||
- `11ad0183b3812c612e30eea9caa754057bf827bb62044858e489f10b2dfc76d0.json` (~652 tok)
|
||||
- `122c7f6dd4c6905cae678fec4ec301f801c2a4ddf34d8cd5d114259499e42888.json` (~8512 tok)
|
||||
- `122d2b1601ab8c15ea9aee2e4fae51e0c85d8ec4559088e9ca8dfc2b596e1dc7.json` (~652 tok)
|
||||
- `12670c77f8cc5ed14a8fd970a52efd6443249934afe535ba5c942e20085ad13d.json` (~23240 tok)
|
||||
- `12681440af2141163a9c8e268eb57806917f49c6f9c096cc62916803475ce07a.json` (~81 tok)
|
||||
- `128104484efd94bffee42db38eb7f45f674e211912b614f10ed09b85616a2697.json` (~652 tok)
|
||||
- `12c733acf3d86ce393643e971e1f4940fd1758a2696817d2f2a4247f41b2f480.json` (~5120 tok)
|
||||
- `1308673b6c4853c3a6bffa60799745c5db7ee0be57b2f2217e66a5fefded692c.json` (~1110 tok)
|
||||
- `1312e11eefb522023a6437df3bef6b7a618dc2a482005da8315b46a9fe0264a6.json` (~7909 tok)
|
||||
- `137c52263f62b509bdf82667625be28668f70fcd3b3d560634d481289b687386.json` (~63297 tok)
|
||||
- `140148117c95271d4f0edef28aa39281e12ea2d7b91e5e9e546f5865adabdcf2.json` (~10944 tok)
|
||||
- `143d4b85cb0f06f22469489e94daac7b33824a4ba8aaf2a32ecdddc399a2a221.json` (~15340 tok)
|
||||
- `1459add9026c6903d390de90c7d1dea4d5a9d381a3930785e7a480dc27b5ec00.json` (~6672 tok)
|
||||
- `146294343cf8e6ac9b9dd5fae0a60bb65150f8da6bb5a386bc0da0f02183c14b.json` (~269 tok)
|
||||
- `14a93f201d19d457092b9bf881e8ac62f44caf42d70192b2106944c8380374d3.json` (~76 tok)
|
||||
- `151a099eec558a9e4c89bc08aae88ba096681971be1a94c7625343fd9a3a88e5.json` (~7640 tok)
|
||||
- `15224225923ce6697e098312e1dc8429b2768b415534886f344c212446813d83.json` (~495 tok)
|
||||
- `152d9c2445a8247d38b52f6ecab0cf6f36ba144aeb93e0bc7603445e7ed2c6cc.json` (~5075 tok)
|
||||
- `155212874dafa331b4fd417f7ebd9bafae70754b134e49c5cfb694a47c21def5.json` (~81 tok)
|
||||
- `157041af18754d4729c7ead7ed02190c14fa052b3c65deac29309014894e67a8.json` (~10023 tok)
|
||||
- `157c7929f5a2274e2f98c623d21343ab2f01711d4fd4e742caef92e2bf2b3463.json` (~4857 tok)
|
||||
- `158090133ac47bc62dd65c9b1ff44bdc089811c93f672d85e0711a994b41af17.json` (~8238 tok)
|
||||
- `15cb56feba56942d333a3d530b675dcfddd2cf20fbd9895f6a45278e9b07daa9.json` (~81 tok)
|
||||
- `15cc2d188fbb7c7bc937a39311efdc8cd2536b01947309e45f5563c0e40a7923.json` (~1376 tok)
|
||||
- `15f43122fe941435ef4d1c92bf796b0198998b92df0b915e6e173a4a0d3e0dcd.json` (~2943 tok)
|
||||
- `161d1766fb319e69d5a2a0b8ae6b0fe31094664d773a6bcabf946ea47a40960a.json` (~81 tok)
|
||||
- `16307f083b22977db26185b0f7407bc40e4f74e2098040ae978316d99d4444d8.json` (~335 tok)
|
||||
- `16831fd1cef7d7b478af79b83f1cc33f276badb9f9bc9f20c827cc88ff417cff.json` (~10294 tok)
|
||||
- `1776308554d389fe728a6b9a5c091cc650242a088a08818acaba4aaef16ff4e6.json` (~81 tok)
|
||||
- `1642d4ea0bbdbc98ef3abccdbc361010f2f63d60d57d01adfd5ce1ac487809d1.json` (~10462 tok)
|
||||
- `169ccd05e20660f7947e1eec4a97149bb00203e4adf8525bbbf558c394af8203.json` (~81 tok)
|
||||
- `171c864c6be7b44ddbc4207c8890610c7cff16a6d109e83f1ef1cf872cb1d5cf.json` (~18765 tok)
|
||||
- `17315b1327a8f2b3df09bccacb57ea8753c011c3b898c7527349b5d113841b82.json` (~10352 tok)
|
||||
- `17c6bf76cfaf2951c6567ecaba00b057c5bf1a8c380a669df83e1a3cbecf630f.json` (~482 tok)
|
||||
- `17e09a3e268eac887a1cc2b71fa136349aec63609ea2848c6d2b745c6cc150df.json` (~269 tok)
|
||||
- `17f10a996c2294de1c285cc0663c48f43b791ed82eb71ac2cb754bb4d1a93e70.json` (~15467 tok)
|
||||
- `17fd2ab30160b12df998f624152b8f2fb7549f4e7b86c077d074a72810c11f08.json` (~8238 tok)
|
||||
- `1814c8441ce6baf90c83e273914a1f472136a2e348e12e9eaf902367a650574c.json` (~323 tok)
|
||||
- `181955f831ace827cab57775ab5c606ca3546d23e3b084d7bb3a386ab30603dc.json` (~182 tok)
|
||||
- `1870fbb4d9d0c1c96f6d0206b980cc4acaa128a432b581bcacec7f6fe866eecd.json` (~1988 tok)
|
||||
- `1880ee75786ce6a428cdaa0f911f58973080482a358b7886fb6aee65c43beda1.json` (~2444 tok)
|
||||
- `1882cd8ec3f0690693a3f400ebd6902a184ca6f0a65dd476837098df893adec0.json` (~81 tok)
|
||||
- `189fd5992484b9c2f1589b17d6564778263f7dce96697c30e3412d0fad0c5da2.json` (~8238 tok)
|
||||
- `18da646f592197b9cdd0707fbb41cbd46f3a59195718deea292f74812b4bb3b7.json` (~652 tok)
|
||||
- `18dc5928206967173c007cbb765877d41488abd0fcb0de1d56abecbaf3f724b7.json` (~9240 tok)
|
||||
- `19009ec1aff7567353993c6fcb7977616f21f799ba4d63e46047ee80585836a6.json` (~652 tok)
|
||||
- `191b4446d235d84fd3681d7e952ad2552fe572d3a5757a5074074549c4e27d50.json` (~1829 tok)
|
||||
- `19b8b2c7c84599c0cf67fb94d4245dc4ca47aa88b54194aece2463712f9e2269.json` (~374 tok)
|
||||
- `1985ef8f8aff8de336ea7725369480591c8357e1d9cd22fd6472cb41ae61298e.json` (~36811 tok)
|
||||
- `19c8003bcc5314fd08d9000b104fd6d443d7959d5babc40fe8b7eb1397056352.json` (~2145 tok)
|
||||
- `19ef06be285e1769d58acf0223c5a445d872c31d629d30dee1c5eeb06c69ded7.json` (~35085 tok)
|
||||
- `19f40018949a73961f1be70427bfa744ba1dbb6c67e597c65b3f652d357be8f0.json` (~324 tok)
|
||||
- `1a91351d50722bca9da67761028884679cbc357a5587e783ce691d7cab7cf65b.json` (~1075 tok)
|
||||
- `1ac9034146e90e4bd572d5c5bfc825730a5387b7b08e37deb1d86cab034a99f9.json` (~81 tok)
|
||||
- `1acd8bb9dda7c761a55ae518ce15546212e95c0c5e3e9393dba5bc7d15813796.json` (~10462 tok)
|
||||
- `1ad8474a865962b24820ff0d953482ce1a9644e667ba136752e0388d24f0ebf9.json` (~4490 tok)
|
||||
- `1b3b474cc46958f3b63e6ef226c2f6b5d7b0275ca7d4e62a2e92c331fc09972c.json` (~6923 tok)
|
||||
- `1b3eb8209fe8ae83a46fe284c12c8577df135d756ac662156e4a61eab79e5601.json` (~1934 tok)
|
||||
- `1b5a6250958790565da71172c0f2da9c26e4651aaec7a353d34f33555f38f4c2.json` (~1904 tok)
|
||||
- `1b99d83061721291e801fa6f10dc01f973f1941a543a381aff9f1db4aa6ae0fc.json` (~5762 tok)
|
||||
- `1bb7af2ce570f59f08babf055ac4b6599c1f138fd580726f6f8056071d6790e8.json` (~76 tok)
|
||||
- `1bcad025e4de349658f8894bb6b724bc2b9937cb0a888ae13347682770ba2363.json` (~11054 tok)
|
||||
- `1c082dda72fabdaf8c4b3cf9c5891940b0bfba2696c65685421f88592504446a.json` (~12528 tok)
|
||||
- `1c0cfaeda41c83a55365b1b5b340cf80581217b083b93ce0dd156fc886276256.json` (~3131 tok)
|
||||
- `1c1177040b6bbb14eca016acca9d4e99dd2b7caf2522e31222570c960ed79501.json` (~53054 tok)
|
||||
- `1c86ed59f0918f5ea0f47f54917854eb2ce2a9b9b6724a107c9237ed0a7f7ea2.json` (~2916 tok)
|
||||
- `1c906780916fd91e042adaa801f385dc55bc1f29c851908ad61f8c5db0040107.json` (~2163 tok)
|
||||
- `1c9619334c216e67eec7d7a1c420d26476001173e926088e46d4d2acdd52f8d4.json` (~320 tok)
|
||||
- `1d7c4a75171c44fea2a337b2b8756a0b4aefd3407e5a0cce07089b2722f4939d.json` (~7794 tok)
|
||||
- `1dcf7b97c92f5149c264d18f36fb408c96372770c52491ec5d320b4a17533612.json` (~8110 tok)
|
||||
- `1e8313e77c1ec1f11b656b60fdcefe81f0d61322c4cdcbf51cf918c7e7965420.json` (~15340 tok)
|
||||
- `1c96714a7cfa6cd6a3494d1325905a74a6caf801e63a5abaf2a20bb8c8a9e3bc.json` (~16210 tok)
|
||||
- `1d3f7ad969993099744a5c43cd721dd449fffe5670bae4a1e370838cbec89edb.json` (~81 tok)
|
||||
- `1db27410d00e74b35c0e017efa058080cc7641365093dcd76695b333a4dbe052.json` (~406 tok)
|
||||
- `1e6d25b23b6374258f00dcad017b13af96b179f0ab46b57b645edde2351ce116.json` (~15467 tok)
|
||||
- `1ef77ec32c809ea9f6abe455804f5630dc3a9693e633b03068f3546dfeeab099.json` (~3149 tok)
|
||||
- `1f0bff79dc1f1fcb886d13bab79c289cd46c2fe9af46ed22739c8076c4a4f6b2.json` (~10944 tok)
|
||||
- `1f6c6535ebbdcedd95bf44e6b96f07f1234d7ea54cb747d6e113f90ccf4bb7ea.json` (~13910 tok)
|
||||
- `1f7e683af7e73fefdcf14ce40c2ee81dfa8aa113b7aa90b51067d870710baa7a.json` (~332 tok)
|
||||
- `1f9c1094d37b3b77b52d3863f257184175c1f040ed46cfbd75b54598f428d466.json` (~81 tok)
|
||||
- `1fdd469bb6d51958ab2ba15266b745abf7014618c2055cf5b098276ea4a501bd.json` (~62630 tok)
|
||||
- `1fe907805f188b6cf4e490bbab545a31e875b83a02bb0e7220e1b8666ddee3e6.json` (~332 tok)
|
||||
- `20456a4402821d8eeb5ccfb2b27a16965b9d4ed85058de966a1f4894fe8443eb.json` (~9852 tok)
|
||||
- `204755bab645cbf0ae12bb0bf8787de41a1f94e10e2988eff7357b93085e7466.json` (~8392 tok)
|
||||
- `20f4f40d81b0a264990d7aab7acdf82a1b66fb32d803b2b878f056973584921a.json` (~474 tok)
|
||||
- `2155a5a1575dcef9f688750a7ab5aef21abf6857b99ca64f0140ec2d95679d8a.json` (~464 tok)
|
||||
- `21646b4f891df0a3a87bbf8bc22fbed445cf2865d4fa7c6b1c3366b5b5a55b37.json` (~1737 tok)
|
||||
- `217a1a381b51231bf05ab3ee23119b669fd864061aa1f9b9e2446263ff38d372.json` (~40340 tok)
|
||||
- `21b020e96fb39c7dc4f05e58b046551b0434adcb3fb196276a019175b1d189dd.json` (~4422 tok)
|
||||
- `2205e0519599088ede54092b2c13d182833ad49f15a64411c26758d1cbcc30e4.json` (~6577 tok)
|
||||
- `2218f88cc182f9064384f7ef7a3ac49908b837dd2d067cd8dbced90e8d043884.json` (~8526 tok)
|
||||
- `226d09a3ccfa62cc05d1b61facad2fd5f269e9f41888caa8457f1dc846435b4e.json` (~3797 tok)
|
||||
- `2272750056f162f7c74228b5a1cf2324b1299841d24f98d0dd31696682e18f28.json` (~2727 tok)
|
||||
- `22948d27d60794a47f2f93a70f92f6b9bdd22da872b32ebbad4812a1a235832e.json` (~81 tok)
|
||||
- `22e61dc743f906387265eea8a41943b22c9aab0f90ec999145a1a7ba7708b1cd.json` (~269 tok)
|
||||
- `232e0527910c5d5dbfcf861f2c6abd8658a0ed73b76f744d24c5cd24b250c344.json` (~8155 tok)
|
||||
- `236dbc464b50c074e467d53cab78fb8af06c934b191349a006f12477317ddb91.json` (~637 tok)
|
||||
- `237f88b4759f44668ce8cd4941b4231ee77fcb94c932ef3f283e596de3de41af.json` (~1262 tok)
|
||||
- `239ef73fc0ea3be8c6ce97517eb62f2c217a5c8b0cbd0c451fccceaf0ef89c47.json` (~8329 tok)
|
||||
- `23ae45376874ad36b52972b9b9031b66aa76f34d1de4b793801908d2dc0bdbc4.json` (~4128 tok)
|
||||
- `23b8085ed43aaaed2d876c21ecb494fa28af40a1cefaffcf5ffc4b131d9ed3f8.json` (~1662 tok)
|
||||
- `23c91fcdfe60f197a54ba90e48faeb3085a488c396aea86acda4c22f2777bc68.json` (~8238 tok)
|
||||
- `23cbeb01ae9cd9669fae81ddb2b2d6c7f89abd17b12365a5f6b256a32a717616.json` (~490 tok)
|
||||
- `24272ebb7e242bd439e42b571de9c4b67ba0f5a3a09c07bde630b6050cbeef4a.json` (~15340 tok)
|
||||
- `2413854cfac3759faf39eae3f1132cac20fc8e916297c82256c957526bfa12df.json` (~53054 tok)
|
||||
- `242faeda0abe77b1e66a515917a267528fe2d1b0c091dbced56e493e9cf69737.json` (~3934 tok)
|
||||
- `2443d5f97a6adffcf84c2d6e593dca849ef0b302414976f1c1f44d4b8b72485d.json` (~81 tok)
|
||||
- `24829855230378343e117d2d2831a59dcb4efc0c8107a76984172094d819a306.json` (~1196 tok)
|
||||
- `2489328d295fdc7968fe9b006fdcf9755239044066224afc4fd3857c3667c560.json` (~652 tok)
|
||||
- `24fa3f5b3b5b6d59088387aad4130a5e2d657f88706205d6dbe57e8afda2e425.json` (~722 tok)
|
||||
- `251ce70fbb55f062faef6259731b4c82238f4f7af22d2c1ed266c86c15263b99.json` (~502 tok)
|
||||
- `25389b2042efd3ef5da1211191cd0c53e4d697dc319451cd5c7289fe6891dda0.json` (~1946 tok)
|
||||
- `255ff40d78e624e0f8c4e96ee7901bec920e078d9960cf9cdafa2a97453e877e.json` (~359 tok)
|
||||
- `2562c246c26f605f6ae96b19b59bfd9d1553b62194d348951b48964429914f8b.json` (~7592 tok)
|
||||
- `25a469841d1a48195460555e7ad298e145ff56da07c945566efcc95ac109b226.json` (~30466 tok)
|
||||
- `25dbcdd9da5297a9d6c98903f9f02d5d9065c88d7299b20c76b01a99f0ea2c11.json` (~81 tok)
|
||||
- `25f6c2a43106baedfadd95c63a6756af97347a44dec2331761762e474aae4efa.json` (~2638 tok)
|
||||
- `260fd344161776959594ebf486e3f15b3e284296c9bce4976df2f3b9c032532d.json` (~8329 tok)
|
||||
- `266573a6c4e7a0f3d1ea12581406c0bbdd0e38321f0573a978db00a9d56cbbd3.json` (~333 tok)
|
||||
- `267c67c5b1f429f43ed99481e42652c822ffcf51d0d0c6ba4121d120c9f74e4c.json` (~441 tok)
|
||||
- `2729cd44ff4befe72984d232b7ae9d0a2bda3c0ad487f7907c73680caec307e4.json` (~14268 tok)
|
||||
- `273726998e57f52d3c0ddf9359e675185f77ff4652c613f53575dc278f326d13.json` (~374 tok)
|
||||
- `27e1c38fff94cb881b9e58c98496e83ae63eb8c025f3872f9a2d81fcbbae990d.json` (~37255 tok)
|
||||
- `284ed1a83c382d8a248273e336c02fb4673b75d877136f1c5f061412fd32dd8a.json` (~17664 tok)
|
||||
- `28773893e82e437c0850281f02e3d0e8c33a905b667ab3b5ec15c0125069cce2.json` (~12593 tok)
|
||||
- `28d052ea57fd7eef39d0ee9fe1d7c7531b0ddea082a123a6997c50f0703f2d98.json` (~1340 tok)
|
||||
- `2930482f7a15ac6d50037b6465a084fdc9cbed52609068751020b5c3016088c3.json` (~1279 tok)
|
||||
- `29cd8d6bcb6dadf0782652207d01f8eb0435d068aaa91c582d98a961fe3ad8fc.json` (~2251 tok)
|
||||
- `29f9d3dbb7661a1fec506165923e621ed0a0c0f1fdab925cb1759bdaf50e5f03.json` (~154 tok)
|
||||
- `2a0380b1dc6fa6fc26764acc5139a7e3de3f3973646f38c58141c13f958bd090.json` (~10023 tok)
|
||||
- `2a2bb41a3bfba155bd19f74ab5cf8ed888210b7ecc7a11b5be1dd100774bc897.json` (~2825 tok)
|
||||
- `2a3d257e37d41ee27ac06bc1b28716624b811a1cc6446f1e56e78a7086d6efc3.json` (~12593 tok)
|
||||
- `2a5dc701edf8702bf74f2c9127b4fd7d34b53e232f21c275c49c6e89e427e606.json` (~8238 tok)
|
||||
- `2a7e32f31ea531530978500ba347d3eb463270ec6c4f4b0e300c84a9fca4729c.json` (~1131 tok)
|
||||
- `2aac8940fc2b352d82c8dd4f713ebfd622cd4362def8801096016bb713b9f35e.json` (~19676 tok)
|
||||
- `2ad72e7a427f8046c048381f2292e12f34926b58456a02043b407d6cf490a77a.json` (~28558 tok)
|
||||
- `2b0b0f1eb49df36a2e8fd4f88317124dd1edf9a0f08545a1a53accd6b47b16b1.json` (~309 tok)
|
||||
- `2b9e4e051f5e39e6df4dc1c1ff7611d9a3590262f41b88e978fa664c596524b8.json` (~269 tok)
|
||||
- `2baff6d343f2ec98cec20237f085bb8804623059aa145279f05e1d921aa1f8bc.json` (~68072 tok)
|
||||
- `2bc3ed1bf9f42caac3f6b63f8e317f329f2705320f02a330b243e1b605c6bc9b.json` (~76991 tok)
|
||||
- `2b2444e22dedc796df87ccc78174df31031a529b9f3f346abb0e6b2f6d4fc52e.json` (~81 tok)
|
||||
- `2b36cdbba90a7e8d28e41d1db13968f9835bc5eef704899c398cd23b04abcdbb.json` (~1122 tok)
|
||||
- `2b3923c904abf48e7056718e593e92e6cac3c38606ded4c39acbad3468cdd6a8.json` (~8010 tok)
|
||||
- `2b6cec7f7a4d2ff30575fb5c744d62b40821119a034d3ae76babae799ce7917c.json` (~81 tok)
|
||||
- `2c220f2115dcf1eaf97111f87e953d7fd6c1c6ba4ace861b1f7f6b533a8915ec.json` (~652 tok)
|
||||
- `2c248a9563703f47faee7ed52cb4054689e07700bc17f38c025188351b860d17.json` (~81 tok)
|
||||
- `2c4427908596fd2a9eb4fd805b1d8b1f9c825dd2df14b560b1264e66eaab9e7d.json` (~15340 tok)
|
||||
- `2c46e2714bddfc44f5f42657d1bf834a1b50130847ac5742955e5dba3b6bc96c.json` (~76 tok)
|
||||
- `2cbca2b1bf0345399a27e19131777f4d537a479e5d9eea2aefe60bfa6f0c848d.json` (~5279 tok)
|
||||
- `2ce08b7c8b08e1091ac5949fb66ae92fa683e1be76fbaf7b41b958dc151441e8.json` (~24408 tok)
|
||||
- `2ce67e45b669a308d17c7ea9c19a75c9ff4e6b8fee11e4a38b9c52f35a1cefe1.json` (~81 tok)
|
||||
- `2cfa572dc6e7e65215e965583eb1ed589b5e67236b2ba727a7144ad8c09e92c6.json` (~269 tok)
|
||||
- `2d2ada19437fe72674471be93be30236721422b4f63176a58b6470f26ba5d88a.json` (~81 tok)
|
||||
- `2d59df43c2cc4d09ea8566c4a94d70cf1a59b8d86e8fc4f9d7c49e92fd356d75.json` (~702 tok)
|
||||
- `2da311d70331211f4889d37b811bfe6d4de072f101d561837f94cc46b2442696.json` (~81 tok)
|
||||
- `2e097eddc32d49647dfd7b1c33bb454265b5587ec5bf148029c1ea496321073c.json` (~5588 tok)
|
||||
- `2efd2ac807fbaf51e1d509f8703b3a911bd6a2e99f11b7738f040cd777e02232.json` (~565 tok)
|
||||
- `2f253d9538cf47faa19555fcece49f789b8e0379c5d654a0fb17c2b9037d7e6e.json` (~619 tok)
|
||||
- `2f314917390ab9fc2f54f98e76033a35b0ac32f11d28a23c6cef9b10c03440d5.json` (~296 tok)
|
||||
- `2f57014ed460815e6563369443502723f1348db370e70fbc68644596d7ac3dcf.json` (~10023 tok)
|
||||
- `2feeb0b89a79e107de5bd960ff0928229edd8ecd466be018d2f3e01ec37f92a1.json` (~406 tok)
|
||||
- `305c23f862d15ebe5ac5f8a24faff332bc1b47b912e318e8cf48aa9f8caf6845.json` (~30585 tok)
|
||||
- `30e46776a7b0b61fbf25b89f2cbfc05cce0db668f4e79ff9e2b5094e3b968fb9.json` (~2029 tok)
|
||||
- `30f7481b9127ef4a3254ec2366cdb021459b664d02579f3dd1e68507565b835c.json` (~81 tok)
|
||||
- `310e465ea5ffc9068fb51b3d4fe4023772368a11d980ff9dad684b2c35f085c5.json` (~652 tok)
|
||||
- `310cc0c86b9b4731dc1657feaef6f82609534758ce5844550c1f430587248129.json` (~269 tok)
|
||||
- `31314813c9df83c12d3ce583abfae372cdd2b6fa81c85668871b82ea4a9347e0.json` (~10611 tok)
|
||||
- `3143298f7a6b86ced4d5f90d69491d3330c949f704944c0e74db95652a516dc9.json` (~31859 tok)
|
||||
- `318a72243b566402f62baddf5e7dd68a62f4f41eb20f5a9874529363bc92b8c0.json` (~8127 tok)
|
||||
- `31925f74354626a58d815d51a97d505725f05f1cdd757f6d07c35080d040c3b6.json` (~10147 tok)
|
||||
- `31e922879882c5b3f25d90a04b60bd1d7379d4c9129844b17a6829bc4bf89615.json` (~2210 tok)
|
||||
- `32263562937e5b72d3330814c6f107330176a5270e05f2009f0cb77800d669d0.json` (~73 tok)
|
||||
- `323b81424dba12471c44d1dc784633bf1d5a69a98fe31afafbdc964b63dd28ed.json` (~7914 tok)
|
||||
- `324d5a22da8a2bc8a6a657073dcd1ca83f95aafde5553a4425c41fdfd18b1502.json` (~844 tok)
|
||||
- `32526ddea34efdc2b69417295f72211dd5610b79379991f1bacc52a8e41e2bc8.json` (~5167 tok)
|
||||
- `32b58522b3e3c7218eeed3098b6c5c4f877195b74425407770722c3dd4e2aa8e.json` (~1988 tok)
|
||||
- `32c297b24895d5f2897691912f3881bd9390917db087a19c1e8d72b66e2060c5.json` (~16691 tok)
|
||||
- `32e14fb903c56f7793ccae75f5dc2c5511d97225d96486afd56604ecb3a43b7e.json` (~81 tok)
|
||||
- `3345b676a5c6e6c2392fc43615a0e0da1824dd13c83d3598598690f8c63697ef.json` (~15396 tok)
|
||||
- `336119f075fb4d93e7e000da36699438170281aeb6626e539422c74dbdef4df4.json` (~36810 tok)
|
||||
- `3364f257aea55647d0ffd26f5c508d04e0cbf24c96c9d1090eeade816bd1adb9.json` (~5386 tok)
|
||||
- `3399458fe3fc893d4f104368ef5ecd4afb95871b46436b2d09d9c71e700cefd8.json` (~785 tok)
|
||||
- `3484ae0eb95d4002544aa8cf72d51923c87611c889de4d867fa3a4f2a24fc153.json` (~8120 tok)
|
||||
- `34a0845d9715d767248d397f0f2234418c0a1828a171bccb04ffba3e1929f668.json` (~652 tok)
|
||||
- `34a753617eaab17b4b08bbf3025b584347ef95d7cda711babde228191f653647.json` (~9568 tok)
|
||||
- `33be867656840f3aa175ca57d8ae13a9f8edfb83c35e00dd876d55fe3d908d7e.json` (~15467 tok)
|
||||
- `3410dfc1ec23e9b3f27898e5d1ad8b821da4176eb9d5591666976699efb3126d.json` (~4145 tok)
|
||||
- `343a8094773eeff36246fa33f6bfa7ec36ae532abb4da401e808928adbe70424.json` (~15134 tok)
|
||||
- `344f4897098b23fcbfb47fdb64e4daf881dc02570068ecf06b077259f63ba525.json` (~1005 tok)
|
||||
- `3474f93a95d80c3c6cd6d95cafb0f5e6b66615e9c1aefd379dfe59e9d229d353.json` (~592 tok)
|
||||
- `34ec637d899925d6db1ba12bc8bf368183f2b0e3636ba7212b2056bf7bd642c6.json` (~85 tok)
|
||||
- `35084efafb380fce518784bf2d8b13c3683acabd3190810991b97a34c52418c2.json` (~396 tok)
|
||||
- `350898ae8fe9d6a099d42d9d292c43efb8a1a37182085f7acadb6b34602d6024.json` (~11199 tok)
|
||||
- `351d8046e5f9f289979f6864f99509fc2fc25b015c2ad343ab4b99193132e56e.json` (~715 tok)
|
||||
- `354f4cfb8682d9b8765e7580be9c5bd1555799633a6d59dc3aea0150f0528acd.json` (~15340 tok)
|
||||
- `35687c768af0e0ff77d76bd1dcc018cd30b6b20c597a67586137ed0a9878bd61.json` (~965 tok)
|
||||
- `35795db2910ab61ee7bf5537a6ca93f93732832dd4b43de3f36720d386de5fb8.json` (~1005 tok)
|
||||
- `35ec72e792b59b19044577eb54a5e7540f026ac35c6003306f597a5f1387d227.json` (~5448 tok)
|
||||
- `35f823871a12ed1809e20d70a663c8a8929fb35f3748705499e32df6cc4ec5c6.json` (~277 tok)
|
||||
- `35fd2a3ceb004b1d47fe5ccc509be97f4d380faebcfff32399237e9dfab59c00.json` (~6584 tok)
|
||||
- `3647cdf8214f8d719efeb55483edf1e5f601279994194bbd06e8ca067a55c2f7.json` (~108634 tok)
|
||||
- `367e004bbee4bd3a32e05eaf817f2dae69e39266c36b1649e4d3929362e383da.json` (~930 tok)
|
||||
- `36fa47b81d63c37de1102693c833e5a1c113dcb414837b3643ba68ff94058545.json` (~592 tok)
|
||||
- `36b96cc86a02e341b8b895b48397dce2385a86221fc47109ccddf46daeeea250.json` (~81 tok)
|
||||
- `36bcc986a2db0fbc0ec8b96ac0cfb16d96043e0f845992576d762e51fd7b242e.json` (~15340 tok)
|
||||
- `36c04caec72b0f97b36069c3908d6192104c00f4d704bf0690fc61765531f523.json` (~10462 tok)
|
||||
- `3702a2af17c73eed0470df46cecddc0b4c9a93aaab893ff5292d5c5b9b4ed512.json` (~9636 tok)
|
||||
- `3714706b1961d4945195e8613ba7a5fcbf550c040e671d5860f2e8d747dc477c.json` (~6708 tok)
|
||||
- `37a543436eff088729eaa2a790c9ef9e58f59f226122e12aa2a0e6c38bc6f5a0.json` (~1870 tok)
|
||||
- `37cc6559d16e5a27798efe47d8ef040ddb31d3774fd6ef9e594793d8083e99f9.json` (~15467 tok)
|
||||
- `37cfdb1b9139384b7afd6b71a19606306bde221b63a2c4b3eb65ec5307855e45.json` (~17007 tok)
|
||||
- `37d60444e71080df18fa511b971b5c8d3c62ac5e8cf74527dcdab21a2ae69a76.json` (~561 tok)
|
||||
- `381ebaece388d903a5945752e2c6a1b923416e6e1f5373b13b903bc69639919a.json` (~6344 tok)
|
||||
- `383a6850c6e5fd218700f04e4f3fa5a8d616ed90af9f0ce63cc9fbbd828ca42c.json` (~374 tok)
|
||||
- `386ae1e8bf78901053b32e9e09acab3017e1b15c172bdb9a7f0ba752ff25583d.json` (~586 tok)
|
||||
- `388dd55aa581a70dbce06969cd866d8f46cdfaa55cbe211302cf06acdc9fd64e.json` (~2999 tok)
|
||||
- `391cd94df914760e40d43aa047d32a6105918d3006e76f829cde3d5e77ebd1ca.json` (~652 tok)
|
||||
- `396e466ffa3cc396b083d8500b44a94b3d48f18866394c5d7cc8d1e2488bfc13.json` (~801 tok)
|
||||
- `3993a289fb925bac9ad356f94888ff857a44d592cf7d4c1815c3702493f24568.json` (~9385 tok)
|
||||
- `3a08cad607debb49f322e478b38e10607c1af8bdda7e250c1c1a9600ee1eb05a.json` (~269 tok)
|
||||
- `3a1c0a4d790a5bb1e032f9fb3e075ae112f12ebdcf0d096e53d65eddb606120b.json` (~300 tok)
|
||||
- `3a38eed59cea97b4e4b01fc4c0ca709a8a1e5c2065d6313ba025dc2dc239334a.json` (~81 tok)
|
||||
- `3a5d7638d9686a803cdaf98d5696e7dce79ce8876e7dad5a4f4a9cd9b26f473b.json` (~14751 tok)
|
||||
- `3a68f18d4842f9b82045fc5d795559e4763a993fb90a5115e8b0a941a2c4530f.json` (~15467 tok)
|
||||
- `3a6a97b8b8e2a07a3727870d8d4cd2bf8e52b8c30bf792aca0ac507bc99df405.json` (~9546 tok)
|
||||
- `3a6b9bf4c71e1e1b73af9874b2b0527602a93a58d92bba1df2e023b12827fc05.json` (~31127 tok)
|
||||
- `3afa3ebabd95e2dd6a480b94ee3fb361d3662421e7093c382a18fecbcae3322b.json` (~269 tok)
|
||||
- `3b3d41a9d9e7bf1a198634ee72bea3a02b87747fa073455ea33e203dca82b5f8.json` (~3990 tok)
|
||||
- `3b69e25c707eab4010cdbda8e2318a464bea4be6a2afbd7a985f7ea7706e09d6.json` (~2171 tok)
|
||||
- `3b768831aacf2f624535f6b5a9fcfa16db0b0650647dc2f7cea8d8535e708db1.json` (~81 tok)
|
||||
- `3b7d7fec7c26be70ed24ea69bb6ccd4bd5283f2cdab0bebee83059abce35df97.json` (~81 tok)
|
||||
- `3b9ea86d334e477b40b39d49f89f56cf27dd4797dce0436bfda50f157d20ca97.json` (~592 tok)
|
||||
- `3bc0e6a7a00dd29f382b9c836dbb598929fd4e4edb1b450959d00c8e4712bd8e.json` (~636 tok)
|
||||
- `3bccc4b1e70a15f6411c964ff222a7ec3485bd8599710bfc408b3420fa03469f.json` (~10023 tok)
|
||||
- `3c3cccc9607f4041ae193d3d0447f495c08dfa09f28d9ff75b4f73b98264d0d3.json` (~725 tok)
|
||||
- `3c576da7137df22cc5fbd982fd231d9d40d374a77c7d618081ba7018f5eec36c.json` (~4654 tok)
|
||||
- `3d0a807870ec7f6703eab25022ba70e6d12cca273ef820ea6e3bc8a31c97afa5.json` (~10333 tok)
|
||||
- `3d7e3a102cd4fc3bc27f599364f8f63ec75e6cb1ea3c47bcacc0e1261b485a15.json` (~3783 tok)
|
||||
- `3da9ee357a09d89a26df59bd0393606349c2d0849078091078924f60b5cdfd51.json` (~495 tok)
|
||||
- `3db60e6e67d5bb6410285138cb4c838b8adb1d211ebe6ee0c4938820fad4c66b.json` (~269 tok)
|
||||
- `3db8027a5cdc79633520de23a59b83fb76ecfba3132ba8a4f52e785220157e70.json` (~336 tok)
|
||||
- `3ddf0434ea5d953052cfdb1597b14704a3692537f86a9fc442ce21024160eb1a.json` (~30420 tok)
|
||||
- `3de5f25efab468345f714829408944025550d1056ba2b46fa3f3f29bcd5a9573.json` (~2258 tok)
|
||||
- `3e550bfbc558f9bcf5301570b291b375c3b2218478c6fee87d712bafb942dfe7.json` (~86150 tok)
|
||||
- `3ebb59da3bec8cfea2f9a426717e513a696eac7f6e08672e0a2d7f72ac811ba6.json` (~23800 tok)
|
||||
- `3f262362589f503db1e19a46c668f044429f325be927582cda524ff546e3cd21.json` (~652 tok)
|
||||
- `3f48feec3b4d3fb5184678ab68e9b1584d493ec8b175ad30122dc7058d8faaa5.json` (~81 tok)
|
||||
- `3f88bc2442422df4bf6036c3b2a1b3409779cbbeff5df7fb2c031bd14f973119.json` (~6809 tok)
|
||||
- `3f8d83e0da73654fdad2d36d37d926e0fcf87f3fc645834c75938afed280aa20.json` (~81 tok)
|
||||
- `3f9cade10a0070913a41048bc2462ee7a4680399174c61c30c8d6e8a6e722cee.json` (~4780 tok)
|
||||
- `40138b45926e46347669dbe289bb2ea7597aedc0eb558bd0d39aacfef5df62e1.json` (~2333 tok)
|
||||
- `406549c33ed009f33fce2953bac23493fac1a3b8ddbb7f4ededf56d4d4467d29.json` (~10135 tok)
|
||||
- `40a5802d6063d7dceabcd49010eac965e03377546e69b2839ce06e3bc19fb99b.json` (~3592 tok)
|
||||
- `40a70514720bf07cfbe92de67954839e91019b1170af380b1570f710e739a49d.json` (~652 tok)
|
||||
- `40a727896bbebc2f6e00afb878b8c666f7fdefa3f34fea7215de4ed042087b96.json` (~81 tok)
|
||||
- `4109287f426902eeb3615a1912430eac8ca65558cc4ec26c781ea237d5460ef0.json` (~28018 tok)
|
||||
- `41bc353adb8853621ebd82fa735c038f98fe3a7e14d44a2eba6a50f01b4cc944.json` (~1511 tok)
|
||||
- `41e4d75d97d06c3f52e77585e5d114ccd5a9ac8e21767a2b7c74cc568dc7b5b0.json` (~298 tok)
|
||||
- `4244574967e1be57fe35ab1b68d8b31aa262d50a728d8877e7af61f9dec09cd5.json` (~526 tok)
|
||||
- `42fc26ddae39455e27588e5cd8b7a61961a4e5ce52a4e6c77fce30fd813ebe38.json` (~465 tok)
|
||||
- `431735fb57e4e50e282f8cc57b09d553ef1b88f135996cfd253bfac5ff186bd3.json` (~2752 tok)
|
||||
- `43ba72a48acad7a07213454f6c1f78abccf9dd5ecf463d5863b83774dd99bf49.json` (~1010 tok)
|
||||
- `43cca59d292abf40819e07f2859c2dce727ca71a7334ce8384e9a1c50d922337.json` (~3604 tok)
|
||||
- `44245055f1f2aebb7db03f3ef508b06f9357cc6f5671308e8bfcaadf913972ad.json` (~2335 tok)
|
||||
- `447da620ccec613360eb8b68fa18e1605d9265e6469f8ba3f9d05c36292db4be.json` (~652 tok)
|
||||
- `44d133b11103df5a5455f0e124056054065a43a4cf9bf6608a157d98a8247919.json` (~2414 tok)
|
||||
- `44eacc4abbec955dd4031ac82e902ced758e0fc319af2c17d2cc650d7c8701d3.json` (~648 tok)
|
||||
- `4569060fc750bc7f993cb0eef888a341c4b9369a16cdfff680a749a2d8eb7679.json` (~14232 tok)
|
||||
- `45a3f01bceef855fff2282e1e2f5a96edaf6e49eb2dd6774de2c951190939476.json` (~876 tok)
|
||||
- `461865ad5d4eb698d9f8b0d0189052bd8733835f159fdec76ee474deb74fac89.json` (~466 tok)
|
||||
- `468a56306bfbb4b7f8f052404e6f4d239bde83c10b206520cb8e9fd6631647c4.json` (~1038 tok)
|
||||
- `46caff82b95eaec8f55497010d7bd131f4a8651ee883a4d3ea1a121cfe515618.json` (~81 tok)
|
||||
- `473aad72b667e57806070866e41e43c658d0c08d9422b2bab8641b027e29d2db.json` (~53573 tok)
|
||||
- `4761c9bb00e54bf17b43a2ebb04ba21b531111c34e93a9dbc957756076e78bf8.json` (~25212 tok)
|
||||
- `480dbaa39314c68c3d4da3c1cb184846dc98ea3f672516afb1683ad674bc88cf.json` (~678 tok)
|
||||
- `4864f62b6a1943ef81d8e4f903830c54ca1451bf8656bbcacf0b603e9ca39af9.json` (~238 tok)
|
||||
- `48b009bc246cdc46da82cb6eede2d25cbfd574593e06dfcdfb4d29d67494094d.json` (~27795 tok)
|
||||
- `48e2a62af809b08d550c4c38d6312ef86dddbab71aa13f2b6058f83da4c364eb.json` (~1226 tok)
|
||||
- `48f612d06f7f9ac59bc69ee5cd647248f72809eb7cb81546a1d7fefc9a0c560d.json` (~16313 tok)
|
||||
- `491112ef541e46590f478bd7be4a1558e96f76b393b4cdb60d70c4335aad80dd.json` (~15340 tok)
|
||||
- `495102d8b31e92493be8301b6d6af618f217d67f5d23eb73970ed03e4e6aae38.json` (~2679 tok)
|
||||
- `496de07d7e51cef1389ef84f25304ca54e6ed1d409b90c924d73cbf88230066d.json` (~49671 tok)
|
||||
- `498740b1cb5104685bf952a5bc3003a35ccb17d29fd01ab233424e7550a17bfe.json` (~41420 tok)
|
||||
- `4a16de26cbf3a4184dc0761323b9a08875225fe03c549d151fbc746db1144a7b.json` (~5507 tok)
|
||||
- `4a1af2b057f91ffb734e4dd1c6e02a70f7054d58da2611f6ff04b74883f2c46d.json` (~5486 tok)
|
||||
- `4a48ff8b5aa860c94ce2439538aaf0a877bb759fe3f312b9a3be65dd8d121b47.json` (~3390 tok)
|
||||
- `4b1e8b3876352a607a2d7f8945f9770d648a45f3285f916ab81b29798e1038b9.json` (~5468 tok)
|
||||
- `4b648434f378dc77cabcf7ad726d40ab8a3ecd4bd96c25fe2b9b303713b85cfd.json` (~71366 tok)
|
||||
- `4bf3a6240c3116e762e6c911280d0bc5e14d68dbd7b82caf61f8695a11587e72.json` (~1277 tok)
|
||||
- `4c776cf52b544a88c4c9a52d25dd1520c7a30cec47d7e0f08f0acb0eab7348cb.json` (~20902 tok)
|
||||
- `4c8845dd0c079a771cb7e6ff15708701de63a89516b79a1ec9a9cc9a2a5a4801.json` (~486 tok)
|
||||
- `4db3c1de6ac63ea0008c2a929d31af6a328b02f1f9ae4cca67aaf9e74d735e2b.json` (~2641 tok)
|
||||
- `4e034b866804145f6dc85a7dc7fefcd38219624f1b4ee747ee6a6ac1f40b2644.json` (~6371 tok)
|
||||
- `4e1c9fda5ab7f3f66527cd52ef0fcc20daea535347855acfe4086cd0f9dd10c8.json` (~1221 tok)
|
||||
- `4e81d1fe6f1cfc63de7e1a59695f9687deecf540b45fa7df24141de82335a8b6.json` (~3036 tok)
|
||||
- `4e90715ba4f6253b1220ccc8949da57080ace7973196ecb257522a26b6288952.json` (~626 tok)
|
||||
- `4e96707af64de170915568f441d41a15fe7aded395b3024dd8d2c13d83fe38c8.json` (~10023 tok)
|
||||
- `4ee847c53c563a13117c4c6c9079895077ff7d18e951e8860530548e82f5f3e8.json` (~888 tok)
|
||||
- `4f1695f88fdc020f547c2cf509e08b8facfe69919b541b554c0aec76ec04537e.json` (~1432 tok)
|
||||
- `4f385d090c5ace09ef00e7883f4d965d6212218ea58df3f78088208fa47c7d56.json` (~8132 tok)
|
||||
- `4fbcc8c4f9a4190c3d5e6717b7d8315428abbdbc24b390b1081ad03954769178.json` (~181 tok)
|
||||
- `50b5f4dfe98634fea8d756c15c1b3f4d86fc1d607698133430210d0d97977856.json` (~1479 tok)
|
||||
- `51103bc49cc87487f9efa30dcffc5f7639f0cc6cbf8acc722f5da33fd0ce32e3.json` (~8308 tok)
|
||||
- `5175f1392d190fb958eb902f22aea50b3506648b739e0cb8a7bc42d389231e4b.json` (~4502 tok)
|
||||
- `51d646ce7e4c76a33d29d45d20149ff8825fe4339a842c2d000ed93d9e46f3de.json` (~12490 tok)
|
||||
- `521cf1ac969a070d624e689274f75a76fc09d84e8a8ba86908b5a6178e776b03.json` (~10023 tok)
|
||||
- `527377e036844617174eac5b08965176bb127edc5db3d5a77f362b034a4ec4fd.json` (~10319 tok)
|
||||
- `5286722a3fefdea557784ae42c65304cf227e92c44178fff83b9f2394c57b2e6.json` (~5453 tok)
|
||||
- `528bfbd499082064cc9b015a4092ffba50f77e09f2f90b5497e20aec9aba24fb.json` (~138 tok)
|
||||
- `52a31389293007060719822853ae31ba48ea5a86df8441b9c191c64f173fdabf.json` (~2318 tok)
|
||||
- `52eabf2efd0f747f08eaf21143a5472f29e59591207fbc96262b97ff05925e6b.json` (~10316 tok)
|
||||
- `5343869ca34f5c819db0bccc7bc641c28bd99771118d48a495a4df736bc7cfd6.json` (~90431 tok)
|
||||
- `53630b8212623bc229ce74e756487117d202ec192b8120561d9fb5f19012b19c.json` (~1053 tok)
|
||||
- `53c97c6c796b513ab01170b56e6ace6f8725efa934165c6a84249fb0e278b687.json` (~4856 tok)
|
||||
- `53fb90c7ba54ed15ce868aec27ac9cf045a8b40311aa80dcd8c0c27619bb68c0.json` (~437 tok)
|
||||
- `541e0d5c8926ed331444c25f25649063cfd94a800e7a30c5ee42b98e5901e6cf.json` (~7036 tok)
|
||||
- `546ac5f8e828ce834dbc0fd2e1af00b3970345404390f8e8de9d656977fb2761.json` (~2372 tok)
|
||||
- `547c8859d3083d2aa3df826d7d45b335b63abd89ecc10f67810628b642b26f28.json` (~32474 tok)
|
||||
- `54bf081b3ee033f99f334327084ec909aee9a0f93c01e9e7140a6da2e0e218b8.json` (~427 tok)
|
||||
- `5598204d609251195af1fd249f50dc13954de3767c7b1f6d3a5d7217f44ed180.json` (~1398 tok)
|
||||
- `559977e3e74ccca3b7025f5db54827525173610f9ede460adf9f8bd7dd5f5d8f.json` (~60916 tok)
|
||||
- `55a658f23a6ae2ae434c1847a8ce4e67d6077bb1d86cfdae1854e86b332c214a.json` (~3305 tok)
|
||||
- `55e566c3da1711786075f53bd5b5cf55c8c67385305c4e056e5c536471387fae.json` (~3852 tok)
|
||||
- `568a9d16b08a8d2a064b13e70b3a7b1759a7925428815a8b188d8d7258b92ed2.json` (~10763 tok)
|
||||
- `577e37e81e94ff718f63cc8d59fb255f51ad4090898b88e7996ad50eeceb3379.json` (~4531 tok)
|
||||
- `57d9e640d91abd2f583860f3d63b54c0da0d89622194e945122d238869af3579.json` (~2262 tok)
|
||||
- `581c504c4d3fe6ca908dcfdb5bd9507b73f9b821e96fb15d58356ab6f2d04ccd.json` (~2270 tok)
|
||||
- `5939de23768f07fd1a2a227aecc4411f32191e003383cb526e4cafb702571cca.json` (~269 tok)
|
||||
- `594aedf253eb6263edd24ffc2d34c93a770f4bdc0b728515eb053324773d4ddb.json` (~8323 tok)
|
||||
- `599bb763d7c10ec93e8b7401678e30afa6d94d11653c70c46f8c928191a9001e.json` (~592 tok)
|
||||
- `59f075695e0135dde52ba377bdd55298ecb1357cb177b5438d56d4244ec167f1.json` (~2354 tok)
|
||||
- `59f7956e28ec4fc2d018f85746017fdcfa3b6a124bbe9e6b2eb4cfc7e77b9be2.json` (~11412 tok)
|
||||
- `5a1508eee3b01cfd7586af0f88973dfc8c51ef9bdb8d3d6fcf7e40a61d663b0f.json` (~362 tok)
|
||||
- `5a4eca0415834ed8d9a9d84bdef3b95febd8011ac64b257757890b2d6246a1c4.json` (~81 tok)
|
||||
- `5a51f4134cc78b3e9b72124a8117a45e1d9a97d943172d29ba35c29fd844b12f.json` (~360 tok)
|
||||
- `5a822398dceebb814bc41d8ffb801c51f89a129f9d090965877f7f67782a1b23.json` (~4032 tok)
|
||||
- `5a92249e0d4f9dd34ff89497b4c540e6b84a672a455b0a5be5ca710b94e9eb79.json` (~4454 tok)
|
||||
- `5a974e6340597c0447c72b0d7bc977a668a8da2fac8c90c810e886c319e9bbd0.json` (~2118 tok)
|
||||
- `3ce0572164f9daaf2eee6a108f57bb947fb3d812c5578c7451eed78062e38b68.json` (~8024 tok)
|
||||
|
||||
## src/controller/
|
||||
## src/lib/testsuite/
|
||||
|
||||
- `ha.c` — define _GNU_SOURCE (~43480 tok)
|
||||
- `job.c` — include <errno.h> (~8196 tok)
|
||||
- `join.c` — include "config.h" (~2530 tok)
|
||||
- `meson.build` (~1134 tok)
|
||||
- `queue.c` — include <errno.h> (~1451 tok)
|
||||
- `worker.c` — include <errno.h> (~5124 tok)
|
||||
|
||||
## src/controller/testsuite/
|
||||
|
||||
- `controller_test.c` — include <argp.h> (~1368 tok)
|
||||
- `test_commit_barrier_reply.c` — Commit-barrier reply-ordering integration tests (tasks 3.1/3.3/4.3): four restructured handlers, receipt-ACK verdict pins, fact-retention-across-quorum-loss (~21000 tok)
|
||||
- `test_commit_barrier_rollout.c` — Declares explicitly (~11136 tok)
|
||||
- `test_raft.c` — include <errno.h> (~36211 tok)
|
||||
|
|
|
|||
2167
.wolf/buglog.json
2167
.wolf/buglog.json
File diff suppressed because it is too large
Load diff
|
|
@ -1,5 +1,5 @@
|
|||
{
|
||||
"last_heartbeat": "2026-06-11T12:56:32.350Z",
|
||||
"last_heartbeat": "2026-06-16T07:25:23.733Z",
|
||||
"engine_status": "running",
|
||||
"execution_log": [
|
||||
{
|
||||
|
|
@ -289,6 +289,164 @@
|
|||
"status": "success",
|
||||
"timestamp": "2026-06-11T13:00:00.757Z",
|
||||
"duration_ms": 39
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-11T19:00:00.284Z",
|
||||
"duration_ms": 43
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-12T01:00:00.909Z",
|
||||
"duration_ms": 31
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-12T07:00:00.886Z",
|
||||
"duration_ms": 74
|
||||
},
|
||||
{
|
||||
"task_id": "memory-consolidation",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-12T09:00:00.069Z",
|
||||
"duration_ms": 2
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-12T19:00:00.361Z",
|
||||
"duration_ms": 50
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-13T01:00:00.989Z",
|
||||
"duration_ms": 72
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-13T07:00:00.706Z",
|
||||
"duration_ms": 49
|
||||
},
|
||||
{
|
||||
"task_id": "memory-consolidation",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-13T09:00:00.819Z",
|
||||
"duration_ms": 3
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-13T13:00:01.073Z",
|
||||
"duration_ms": 65
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-13T19:00:00.496Z",
|
||||
"duration_ms": 25
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-14T01:00:00.983Z",
|
||||
"duration_ms": 84
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-14T07:00:00.492Z",
|
||||
"duration_ms": 57
|
||||
},
|
||||
{
|
||||
"task_id": "memory-consolidation",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-14T09:00:00.686Z",
|
||||
"duration_ms": 61
|
||||
},
|
||||
{
|
||||
"task_id": "cerebrum-reflection",
|
||||
"status": "failed",
|
||||
"timestamp": "2026-06-14T10:02:02.337Z",
|
||||
"duration_ms": 122215,
|
||||
"error": "claude -p failed: spawnSync /bin/sh ETIMEDOUT"
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-14T13:00:00.912Z",
|
||||
"duration_ms": 111
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-14T19:00:00.194Z",
|
||||
"duration_ms": 42
|
||||
},
|
||||
{
|
||||
"task_id": "memory-consolidation",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-14T23:45:24.291Z",
|
||||
"duration_ms": 2
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-15T01:00:00.635Z",
|
||||
"duration_ms": 82
|
||||
},
|
||||
{
|
||||
"task_id": "memory-consolidation",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-15T09:00:00.984Z",
|
||||
"duration_ms": 4
|
||||
},
|
||||
{
|
||||
"task_id": "token-audit",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-15T09:09:22.749Z",
|
||||
"duration_ms": 56
|
||||
},
|
||||
{
|
||||
"task_id": "token-audit",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-15T09:09:23.810Z",
|
||||
"duration_ms": 42
|
||||
},
|
||||
{
|
||||
"task_id": "project-suggestions",
|
||||
"status": "failed",
|
||||
"timestamp": "2026-06-15T11:02:02.349Z",
|
||||
"duration_ms": 122204,
|
||||
"error": "claude -p failed: spawnSync /bin/sh ETIMEDOUT"
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-15T13:00:00.498Z",
|
||||
"duration_ms": 101
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-15T19:00:01.072Z",
|
||||
"duration_ms": 80
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-16T01:00:00.585Z",
|
||||
"duration_ms": 25
|
||||
},
|
||||
{
|
||||
"task_id": "anatomy-rescan",
|
||||
"status": "success",
|
||||
"timestamp": "2026-06-16T07:00:00.511Z",
|
||||
"duration_ms": 53
|
||||
}
|
||||
],
|
||||
"dead_letter_queue": [],
|
||||
|
|
|
|||
|
|
@ -145,3 +145,65 @@
|
|||
[2026-06-11T09:00:00.463Z] [INFO] Task Consolidate old memory completed in 3ms
|
||||
[2026-06-11T13:00:00.718Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-11T13:00:00.757Z] [INFO] Task Full anatomy rescan completed in 39ms
|
||||
[2026-06-11T19:00:00.241Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-11T19:00:00.284Z] [INFO] Task Full anatomy rescan completed in 43ms
|
||||
[2026-06-12T01:00:00.878Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-12T01:00:00.909Z] [INFO] Task Full anatomy rescan completed in 31ms
|
||||
[2026-06-12T07:00:00.812Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-12T07:00:00.886Z] [INFO] Task Full anatomy rescan completed in 74ms
|
||||
[2026-06-12T09:00:00.067Z] [INFO] Executing task: Consolidate old memory
|
||||
[2026-06-12T09:00:00.069Z] [INFO] Task Consolidate old memory completed in 2ms
|
||||
[2026-06-12T19:00:00.311Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-12T19:00:00.361Z] [INFO] Task Full anatomy rescan completed in 50ms
|
||||
[2026-06-13T01:00:00.917Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-13T01:00:00.989Z] [INFO] Task Full anatomy rescan completed in 72ms
|
||||
[2026-06-13T07:00:00.656Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-13T07:00:00.706Z] [INFO] Task Full anatomy rescan completed in 49ms
|
||||
[2026-06-13T09:00:00.816Z] [INFO] Executing task: Consolidate old memory
|
||||
[2026-06-13T09:00:00.820Z] [INFO] Task Consolidate old memory completed in 3ms
|
||||
[2026-06-13T13:00:01.008Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-13T13:00:01.074Z] [INFO] Task Full anatomy rescan completed in 65ms
|
||||
[2026-06-13T19:00:00.471Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-13T19:00:00.496Z] [INFO] Task Full anatomy rescan completed in 25ms
|
||||
[2026-06-14T01:00:00.899Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-14T01:00:00.983Z] [INFO] Task Full anatomy rescan completed in 84ms
|
||||
[2026-06-14T07:00:00.435Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-14T07:00:00.492Z] [INFO] Task Full anatomy rescan completed in 57ms
|
||||
[2026-06-14T09:00:00.624Z] [INFO] Executing task: Consolidate old memory
|
||||
[2026-06-14T09:00:00.686Z] [INFO] Task Consolidate old memory completed in 61ms
|
||||
[2026-06-14T10:00:00.122Z] [INFO] Executing task: Cerebrum reflection
|
||||
[2026-06-14T10:02:02.337Z] [ERROR] Task Cerebrum reflection failed (attempt 1): claude -p failed: spawnSync /bin/sh ETIMEDOUT
|
||||
[2026-06-14T13:00:00.801Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-14T13:00:00.913Z] [INFO] Task Full anatomy rescan completed in 111ms
|
||||
[2026-06-14T19:00:00.152Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-14T19:00:00.194Z] [INFO] Task Full anatomy rescan completed in 42ms
|
||||
[2026-06-14T23:45:24.288Z] [INFO] Executing task: Consolidate old memory
|
||||
[2026-06-14T23:45:24.291Z] [INFO] Task Consolidate old memory completed in 2ms
|
||||
[2026-06-15T01:00:00.552Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-15T01:00:00.635Z] [INFO] Task Full anatomy rescan completed in 82ms
|
||||
[2026-06-15T08:19:54.500Z] [INFO] Scheduled task: Full anatomy rescan (0 */6 * * *)
|
||||
[2026-06-15T08:19:54.500Z] [INFO] Scheduled task: Consolidate old memory (0 2 * * *)
|
||||
[2026-06-15T08:19:54.501Z] [INFO] Scheduled task: Token audit report (0 0 * * 1)
|
||||
[2026-06-15T08:19:54.501Z] [INFO] Scheduled task: Cerebrum reflection (0 3 * * 0)
|
||||
[2026-06-15T08:19:54.501Z] [INFO] Scheduled task: AI suggestions (0 4 * * 1)
|
||||
[2026-06-15T08:19:54.502Z] [INFO] File watcher started on .wolf/
|
||||
[2026-06-15T08:19:54.502Z] [INFO] OpenWolf daemon started
|
||||
[2026-06-15T08:19:54.503Z] [INFO] Dashboard server listening on port 18791
|
||||
[2026-06-15T09:00:00.979Z] [INFO] Executing task: Consolidate old memory
|
||||
[2026-06-15T09:00:00.984Z] [INFO] Task Consolidate old memory completed in 4ms
|
||||
[2026-06-15T09:07:55.657Z] [INFO] WebSocket client connected
|
||||
[2026-06-15T09:09:22.693Z] [INFO] Executing task: Token audit report
|
||||
[2026-06-15T09:09:22.750Z] [INFO] Task Token audit report completed in 56ms
|
||||
[2026-06-15T09:09:23.768Z] [INFO] Executing task: Token audit report
|
||||
[2026-06-15T09:09:23.810Z] [INFO] Task Token audit report completed in 42ms
|
||||
[2026-06-15T11:00:00.144Z] [INFO] Executing task: AI suggestions
|
||||
[2026-06-15T11:02:02.348Z] [ERROR] Task AI suggestions failed (attempt 1): claude -p failed: spawnSync /bin/sh ETIMEDOUT
|
||||
[2026-06-15T13:00:00.397Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-15T13:00:00.498Z] [INFO] Task Full anatomy rescan completed in 101ms
|
||||
[2026-06-15T19:00:00.992Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-15T19:00:01.072Z] [INFO] Task Full anatomy rescan completed in 80ms
|
||||
[2026-06-16T01:00:00.560Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-16T01:00:00.585Z] [INFO] Task Full anatomy rescan completed in 25ms
|
||||
[2026-06-16T02:48:16.930Z] [INFO] WebSocket client connected
|
||||
[2026-06-16T07:00:00.458Z] [INFO] Executing task: Full anatomy rescan
|
||||
[2026-06-16T07:00:00.512Z] [INFO] Task Full anatomy rescan completed in 53ms
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1761,3 +1761,4 @@ Implemented sub-spec 3/4 of ha-consistency-invariants autonomously (per-task imp
|
|||
| 00:17 | Edited .kiro/specs/ha-prevote/tasks.md | inline fix | ~16 |
|
||||
| 02:30 | kiro-impl ha-prevote: ALL 11 tasks done (2.1-6.1) | raft+ha+cluster+tests | every task review-APPROVED + committed; raft suite 123/0, controller 563/0 | ~60k |
|
||||
| 02:50 | kiro-validate-impl ha-prevote: GO | full suites | raft 123/0, controller 564/0 (direct binary, no real fail), smoke OK, 27/27 reqs, boundary clean | ~5k |
|
||||
| 00:38 | Session end: 9 writes across 4 files (tasks.md, test_ha_prevote.c, test_raft.c, research.md) | 14 reads | ~53449 tok |
|
||||
|
|
|
|||
10809
.wolf/token-ledger.json
10809
.wolf/token-ledger.json
File diff suppressed because it is too large
Load diff
222
src/lib/testsuite/test_wal_wire.c
Normal file
222
src/lib/testsuite/test_wal_wire.c
Normal file
|
|
@ -0,0 +1,222 @@
|
|||
/* test_wal_wire.c — Unity tests for the raw payload wire format
|
||||
* (ha-repl-hardening task 7.1).
|
||||
*
|
||||
* Verifies true encode→decode round-trips for:
|
||||
* - OPCode_walCatchupEntry batches (xce_wal_batch_encode_raw):
|
||||
* per-sub-entry lsn/entry_type and raw_payload bytes survive the wire
|
||||
* - OPCode_snapshotTransfer chunks (xce_snapshot_chunk_encode_raw):
|
||||
* chunk envelope fields and raw_payload bytes survive the wire
|
||||
*
|
||||
* These messages bypass the generated encode path because their
|
||||
* payload :Data fields cannot be carried through xce_data_t (same
|
||||
* limitation as replConfigPush); xce_decode() pre-extracts the bytes
|
||||
* into raw_payload/raw_payload_len before capn_free().
|
||||
*/
|
||||
#include <errno.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <unity/unity.h>
|
||||
|
||||
#include "request.h"
|
||||
#include "xce/types.h"
|
||||
|
||||
/* ── Helpers ────────────────────────────────────────────────────────────── */
|
||||
|
||||
static xce_peerinfo_t test_peer = { 0 };
|
||||
|
||||
/* encode a batch into a heap io_packet_t (xce_decode frees it) */
|
||||
static io_packet_t *encode_batch(wal_catchup_entry_t **entries, int n)
|
||||
{
|
||||
io_packet_t *pkt = (io_packet_t *)calloc(1, sizeof(*pkt));
|
||||
TEST_ASSERT_NOT_NULL(pkt);
|
||||
int rc = xce_wal_batch_encode_raw(entries, n, &test_peer, 42, pkt);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
return pkt;
|
||||
}
|
||||
|
||||
/* ── WAL batch round-trip ───────────────────────────────────────────────── */
|
||||
|
||||
static void test_wal_batch_roundtrip_payload_bytes(void)
|
||||
{
|
||||
/* Arrange: 2 entries with distinct payloads */
|
||||
uint8_t p0[] = { 0xDE, 0xAD, 0xBE, 0xEF };
|
||||
uint8_t p1[] = "capnp job blob \x00 with embedded NUL";
|
||||
|
||||
wal_catchup_entry_t e0, e1;
|
||||
memset(&e0, 0, sizeof(e0));
|
||||
memset(&e1, 0, sizeof(e1));
|
||||
e0.lsn = 101;
|
||||
e0.entry_type = 1; /* WAL_JOB_SUBMIT */
|
||||
e0.raw_payload = p0;
|
||||
e0.raw_payload_len = sizeof(p0);
|
||||
e1.lsn = 102;
|
||||
e1.entry_type = 2;
|
||||
e1.raw_payload = p1;
|
||||
e1.raw_payload_len = sizeof(p1);
|
||||
|
||||
wal_catchup_entry_t *entries[2] = { &e0, &e1 };
|
||||
|
||||
/* Act: encode → decode */
|
||||
io_packet_t *pkt = encode_batch(entries, 2);
|
||||
xce_request_t req;
|
||||
memset(&req, 0, sizeof(req));
|
||||
TEST_ASSERT_EQUAL_INT(0, xce_decode(pkt, &req));
|
||||
|
||||
/* Assert: opcode, body arm, batch shape */
|
||||
TEST_ASSERT_EQUAL_INT(OPCode_walCatchupEntry, req.opcode);
|
||||
TEST_ASSERT_EQUAL_INT(Request_body_walcatchupentrymsg, req.body_which);
|
||||
TEST_ASSERT_NOT_NULL(req.body.walcatchupentrymsg);
|
||||
|
||||
wal_catchup_entry_t *batch = req.body.walcatchupentrymsg;
|
||||
TEST_ASSERT_EQUAL_INT(2, batch->n_entries);
|
||||
TEST_ASSERT_NOT_NULL(batch->entries);
|
||||
|
||||
/* Entry 0: header + payload bytes intact */
|
||||
TEST_ASSERT_NOT_NULL(batch->entries[0]);
|
||||
TEST_ASSERT_EQUAL_UINT64(101, batch->entries[0]->lsn);
|
||||
TEST_ASSERT_EQUAL_UINT8(1, batch->entries[0]->entry_type);
|
||||
TEST_ASSERT_EQUAL_size_t(sizeof(p0), batch->entries[0]->raw_payload_len);
|
||||
TEST_ASSERT_NOT_NULL(batch->entries[0]->raw_payload);
|
||||
TEST_ASSERT_EQUAL_MEMORY(p0, batch->entries[0]->raw_payload, sizeof(p0));
|
||||
|
||||
/* Entry 1: embedded NUL survives (binary-safe) */
|
||||
TEST_ASSERT_NOT_NULL(batch->entries[1]);
|
||||
TEST_ASSERT_EQUAL_UINT64(102, batch->entries[1]->lsn);
|
||||
TEST_ASSERT_EQUAL_UINT8(2, batch->entries[1]->entry_type);
|
||||
TEST_ASSERT_EQUAL_size_t(sizeof(p1), batch->entries[1]->raw_payload_len);
|
||||
TEST_ASSERT_EQUAL_MEMORY(p1, batch->entries[1]->raw_payload, sizeof(p1));
|
||||
|
||||
xce_free_request(&req);
|
||||
printf("test_wal_batch_roundtrip_payload_bytes: PASS\n");
|
||||
}
|
||||
|
||||
static void test_wal_batch_entry_without_payload(void)
|
||||
{
|
||||
/* An entry with no payload (e.g. header-only types) must decode with
|
||||
* raw_payload == NULL and raw_payload_len == 0 — not garbage. */
|
||||
wal_catchup_entry_t e;
|
||||
memset(&e, 0, sizeof(e));
|
||||
e.lsn = 7;
|
||||
e.entry_type = 3;
|
||||
|
||||
wal_catchup_entry_t *entries[1] = { &e };
|
||||
io_packet_t *pkt = encode_batch(entries, 1);
|
||||
|
||||
xce_request_t req;
|
||||
memset(&req, 0, sizeof(req));
|
||||
TEST_ASSERT_EQUAL_INT(0, xce_decode(pkt, &req));
|
||||
|
||||
wal_catchup_entry_t *batch = req.body.walcatchupentrymsg;
|
||||
TEST_ASSERT_NOT_NULL(batch);
|
||||
TEST_ASSERT_EQUAL_INT(1, batch->n_entries);
|
||||
TEST_ASSERT_EQUAL_UINT64(7, batch->entries[0]->lsn);
|
||||
TEST_ASSERT_NULL(batch->entries[0]->raw_payload);
|
||||
TEST_ASSERT_EQUAL_size_t(0, batch->entries[0]->raw_payload_len);
|
||||
|
||||
xce_free_request(&req);
|
||||
printf("test_wal_batch_entry_without_payload: PASS\n");
|
||||
}
|
||||
|
||||
static void test_wal_batch_encode_rejects_invalid_args(void)
|
||||
{
|
||||
io_packet_t pkt;
|
||||
memset(&pkt, 0, sizeof(pkt));
|
||||
wal_catchup_entry_t e;
|
||||
memset(&e, 0, sizeof(e));
|
||||
wal_catchup_entry_t *entries[1] = { &e };
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(-EINVAL,
|
||||
xce_wal_batch_encode_raw(NULL, 1, &test_peer, 0, &pkt));
|
||||
TEST_ASSERT_EQUAL_INT(-EINVAL,
|
||||
xce_wal_batch_encode_raw(entries, 0, &test_peer, 0, &pkt));
|
||||
TEST_ASSERT_EQUAL_INT(-EINVAL,
|
||||
xce_wal_batch_encode_raw(entries, 1, &test_peer, 0, NULL));
|
||||
|
||||
printf("test_wal_batch_encode_rejects_invalid_args: PASS\n");
|
||||
}
|
||||
|
||||
/* ── Snapshot chunk round-trip ──────────────────────────────────────────── */
|
||||
|
||||
static void test_snapshot_chunk_roundtrip(void)
|
||||
{
|
||||
/* Arrange: a mid-transfer chunk with binary payload */
|
||||
uint8_t payload[300];
|
||||
for (size_t i = 0; i < sizeof(payload); i++)
|
||||
payload[i] = (uint8_t)(i * 7);
|
||||
|
||||
snapshot_transfer_t chunk;
|
||||
memset(&chunk, 0, sizeof(chunk));
|
||||
chunk.snapshot_lsn = 5000;
|
||||
chunk.chunk_index = 3;
|
||||
chunk.chunk_offset = 3 * 65536;
|
||||
chunk.total_size = 4 * 65536 + 300;
|
||||
chunk.is_last = 1;
|
||||
|
||||
io_packet_t *pkt = (io_packet_t *)calloc(1, sizeof(*pkt));
|
||||
TEST_ASSERT_NOT_NULL(pkt);
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_snapshot_chunk_encode_raw(&chunk, payload, sizeof(payload),
|
||||
&test_peer, 9, pkt));
|
||||
|
||||
/* Act */
|
||||
xce_request_t req;
|
||||
memset(&req, 0, sizeof(req));
|
||||
TEST_ASSERT_EQUAL_INT(0, xce_decode(pkt, &req));
|
||||
|
||||
/* Assert: envelope + payload bytes */
|
||||
TEST_ASSERT_EQUAL_INT(OPCode_snapshotTransfer, req.opcode);
|
||||
TEST_ASSERT_EQUAL_INT(Request_body_snapshottransfermsg, req.body_which);
|
||||
snapshot_transfer_t *rx = req.body.snapshottransfermsg;
|
||||
TEST_ASSERT_NOT_NULL(rx);
|
||||
TEST_ASSERT_EQUAL_UINT64(5000, rx->snapshot_lsn);
|
||||
TEST_ASSERT_EQUAL_UINT32(3, rx->chunk_index);
|
||||
TEST_ASSERT_EQUAL_UINT64(3 * 65536, rx->chunk_offset);
|
||||
TEST_ASSERT_EQUAL_UINT64(4 * 65536 + 300, rx->total_size);
|
||||
TEST_ASSERT_TRUE(rx->is_last);
|
||||
TEST_ASSERT_EQUAL_size_t(sizeof(payload), rx->raw_payload_len);
|
||||
TEST_ASSERT_NOT_NULL(rx->raw_payload);
|
||||
TEST_ASSERT_EQUAL_MEMORY(payload, rx->raw_payload, sizeof(payload));
|
||||
|
||||
xce_free_request(&req);
|
||||
printf("test_snapshot_chunk_roundtrip: PASS\n");
|
||||
}
|
||||
|
||||
static void test_snapshot_chunk_empty_payload(void)
|
||||
{
|
||||
/* Zero-length payload (defensive): envelope round-trips, raw NULL/0 */
|
||||
snapshot_transfer_t chunk;
|
||||
memset(&chunk, 0, sizeof(chunk));
|
||||
chunk.snapshot_lsn = 1;
|
||||
chunk.total_size = 0;
|
||||
chunk.is_last = 1;
|
||||
|
||||
io_packet_t *pkt = (io_packet_t *)calloc(1, sizeof(*pkt));
|
||||
TEST_ASSERT_NOT_NULL(pkt);
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_snapshot_chunk_encode_raw(&chunk, NULL, 0, &test_peer, 0, pkt));
|
||||
|
||||
xce_request_t req;
|
||||
memset(&req, 0, sizeof(req));
|
||||
TEST_ASSERT_EQUAL_INT(0, xce_decode(pkt, &req));
|
||||
|
||||
snapshot_transfer_t *rx = req.body.snapshottransfermsg;
|
||||
TEST_ASSERT_NOT_NULL(rx);
|
||||
TEST_ASSERT_EQUAL_UINT64(1, rx->snapshot_lsn);
|
||||
TEST_ASSERT_TRUE(rx->is_last);
|
||||
TEST_ASSERT_NULL(rx->raw_payload);
|
||||
TEST_ASSERT_EQUAL_size_t(0, rx->raw_payload_len);
|
||||
|
||||
xce_free_request(&req);
|
||||
printf("test_snapshot_chunk_empty_payload: PASS\n");
|
||||
}
|
||||
|
||||
/* ── Suite entry point ──────────────────────────────────────────────────── */
|
||||
|
||||
void test_wal_wire(void)
|
||||
{
|
||||
RUN_TEST(test_wal_batch_roundtrip_payload_bytes);
|
||||
RUN_TEST(test_wal_batch_entry_without_payload);
|
||||
RUN_TEST(test_wal_batch_encode_rejects_invalid_args);
|
||||
RUN_TEST(test_snapshot_chunk_roundtrip);
|
||||
RUN_TEST(test_snapshot_chunk_empty_payload);
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue