checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
parent
7d70820aac
commit
20e59d0b42
85 changed files with 10525 additions and 1993 deletions
|
|
@ -81,4 +81,4 @@
|
|||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
6
.gitignore
vendored
6
.gitignore
vendored
|
|
@ -21,3 +21,9 @@ src/frontend/apiserver/tokens/
|
|||
.remember
|
||||
graphify-out/
|
||||
.cache
|
||||
|
||||
# --- session checkpoint 2026-06-21: machine-generated / local artifacts ---
|
||||
.codegraph/
|
||||
.xce-durability-acked.*
|
||||
*.bak
|
||||
.claude/ucai-formatter-cache.local.json
|
||||
|
|
|
|||
22
.kiro/specs/config-upgrade-snapshot/brief.md
Normal file
22
.kiro/specs/config-upgrade-snapshot/brief.md
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
# Brief: config-upgrade-snapshot
|
||||
|
||||
## Problem
|
||||
XCE's state machine (jobs, queues, hosts, resources) is **interpreted against the active configuration**. Today a configuration upgrade does config-push + restart + **per-node WAL replay under the new config** (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state. Because replay is config-dependent (the `lost_and_found` queue/host re-homes entities whose config entity vanished), members can **materialize divergent state** after an upgrade — amplified by any WAL tail divergence. Logged as **bug-857** (identified via the `raft.tla` gap analysis; not yet empirically reproduced).
|
||||
|
||||
This was originally Requirement 8 of `ha-term-safety`; it is **descoped into this independent spec** (2026-06-21) because it has a hard prerequisite that does not exist yet.
|
||||
|
||||
## Approach
|
||||
A configuration upgrade establishes a **full, entity-bearing state snapshot at the configuration-change boundary** (the `(term, lsn)` of the config-change marker that `ha-term-safety` introduces), **embeds the new configuration** in that snapshot, and **installs the same snapshot on every member** through the out-of-band snapshot-install transport. No member re-derives post-upgrade state by replaying pre-upgrade WAL under the new configuration — installing the snapshot delivers configuration + materialized state together as one consistent artifact. This supersedes `ha-term-safety` R7's interim config-push byte delivery for the upgrade case.
|
||||
|
||||
## HARD dependency — entity-bearing snapshots
|
||||
This spec **cannot proceed without entity-bearing snapshots**. Today snapshots are a 16-byte LSN sentinel (`"XCESNAP\0"` + checkpoint_lsn; `persist.c` write ~1068-1101, load ~1280-1307) carrying **zero entity state**. The COW machinery needed (`xzarray_freeze`/`thaw`, `zarray.c:695-757`) is **built and unit-tested but dormant** (`kvmgr.c:297` `vfreeze=NULL`, no production callers) — the roadmap's unstarted **`entity-bearing-snapshots` ("Task 11")** backlog item. That work must land first (or be co-scoped). The snapshot *transport* (`snapshot_transfer_s`: chunked, `(term,lsn)`-anchored, hash-verified, install-before-activate) is already reusable; only the entity-bearing *payload* is missing.
|
||||
|
||||
## Scope
|
||||
- **In**: full entity-bearing state snapshot materialized at the config-change marker boundary; embed the new configuration in that snapshot; install on every member via the snapshot-install path; activation gating (a member does not serve under the new configuration until it has installed the upgrade snapshot and verified the embedded configuration against the marker's content hash); post-upgrade consistency invariant (every member's materialized state + active configuration identical); supersede `ha-term-safety` R7's interim config-push byte delivery for upgrades.
|
||||
- **Out**: the config-change marker / consensus ordering itself (owned by `ha-term-safety` R7); the entity serialization machinery (owned by `entity-bearing-snapshots` — this spec consumes it); the Raft protocol (the `Raft TLA+ Conformance` library series); controller voting-set/membership changes (`ha-config-bootstrap`).
|
||||
|
||||
## Verification
|
||||
Empirically reproduce bug-857 first (upgrade config during the chaotic-multi-kill repro and diff materialized state across nodes via `waldump`/state dump) to establish the RED, then prove every member converges to identical materialized state + active config after an upgrade interleaved with failover.
|
||||
|
||||
## Dependencies
|
||||
HARD: `entity-bearing-snapshots` (entity payload), `ha-term-safety` (the config-change marker it anchors on). Consumes the `snapshot_transfer_s` transport. Closes bug-857.
|
||||
67
.kiro/specs/config-upgrade-snapshot/requirements.md
Normal file
67
.kiro/specs/config-upgrade-snapshot/requirements.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
XCE's controller state machine (jobs, queues, hosts, resources) is interpreted against the active configuration. Today a configuration upgrade pushes the new configuration, restarts each controller, and lets each member **re-derive** its post-upgrade state by replaying its pre-upgrade WAL under the new configuration — with no snapshot pinning an identical materialized state across members. Because replay is configuration-dependent (the `lost_and_found` queue/host re-homes entities whose configuration entity vanished), members can diverge in materialized state after an upgrade (bug-857), amplified by any WAL tail divergence.
|
||||
|
||||
This feature makes a configuration upgrade establish a **full, entity-bearing state snapshot at the configuration-change boundary**, embed the new configuration in it, and install that same snapshot on every member — so post-upgrade state is identical by construction and no member re-derives it. It was originally Requirement 8 of `ha-term-safety` and is split out because it has a hard prerequisite (entity-bearing snapshots) that does not yet exist.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: a full entity-bearing state snapshot materialized at the configuration-change marker's `(term, lsn)`; embedding the new configuration in that snapshot; installing it on every member via the out-of-band snapshot-install transport; activation gating until the snapshot is installed and hash-verified; the post-upgrade materialized-state + active-configuration consistency invariant; superseding `ha-term-safety` R7's interim config-push byte delivery for the upgrade case.
|
||||
- **Out of scope**:
|
||||
- The configuration-change marker and consensus ordering of the change — owned by `ha-term-safety` (Requirement 7).
|
||||
- The entity-serialization machinery that produces an entity-bearing snapshot — owned by `entity-bearing-snapshots`; this spec **consumes** it.
|
||||
- The Raft protocol (term/log/commit) — owned by the `Raft TLA+ Conformance` library series.
|
||||
- Controller voting-set / membership changes — `ha-config-bootstrap`.
|
||||
- **Adjacent expectations**:
|
||||
- **HARD dependency on `entity-bearing-snapshots`**: today snapshots are a 16-byte LSN sentinel carrying no entity state; the COW machinery (`xzarray_freeze`/`thaw`) is built but dormant. That work must land first.
|
||||
- Anchors on the `(term, lsn)` configuration-change marker from `ha-term-safety`.
|
||||
- Reuses the existing `snapshot_transfer_s` snapshot-install transport (chunked, `(term, lsn)`-anchored, hash-verified, install-before-activate).
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Full snapshot at the configuration-change boundary
|
||||
|
||||
**Objective:** As an operator, I want a configuration upgrade to materialize a full state snapshot at the change boundary, so that every member adopts the same post-upgrade state rather than re-deriving it.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a configuration change commits, the Controller shall materialize a full state snapshot at the configuration-change marker's `(term, lsn)` boundary.
|
||||
2. The Controller shall install that same snapshot on every member.
|
||||
3. The Controller shall not require any member to re-derive post-upgrade state by replaying pre-upgrade WAL entries under the new configuration.
|
||||
4. The Controller shall transfer the upgrade snapshot through the out-of-band snapshot-install path, not through WAL entry payloads.
|
||||
|
||||
### Requirement 2: Configuration embedded in the upgrade snapshot
|
||||
|
||||
**Objective:** As an operator, I want the new configuration delivered together with the materialized state, so that a member never holds state and configuration from different points.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The Controller shall embed the new configuration in the upgrade snapshot so that installing the snapshot delivers configuration and materialized state as one consistent artifact.
|
||||
2. The Controller shall make this upgrade snapshot supersede the interim out-of-band config-push byte delivery (`ha-term-safety` Requirement 7) for the upgrade case.
|
||||
3. The Controller shall keep the configuration bytes out of WAL entry payloads (the WAL holds only the `(term, lsn)` configuration-change marker).
|
||||
|
||||
### Requirement 3: Activation gating
|
||||
|
||||
**Objective:** As an operator, I want a member to serve under a new configuration only after it holds the matching materialized state, so that no member operates on mismatched configuration or state.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. While a member has not installed the upgrade snapshot, the Controller shall not let that member serve under or activate the new configuration.
|
||||
2. When a member installs an upgrade snapshot, the Controller shall verify the embedded configuration against the configuration-change marker's content hash before activating it.
|
||||
3. If the embedded configuration does not match the marker's content hash, then the Controller shall reject the snapshot and continue retrying rather than activating mismatched configuration.
|
||||
|
||||
### Requirement 4: Post-upgrade consistency invariant
|
||||
|
||||
**Objective:** As a client and operator, I want every member identical after a configuration upgrade, so that an upgrade can never silently diverge materialized state across members.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When members complete a configuration upgrade, every member's materialized state shall be identical.
|
||||
2. When members complete a configuration upgrade, every member's active configuration shall be identical.
|
||||
3. While configuration upgrades interleave with failover, the Controller shall preserve the post-upgrade identical-state invariant across leadership changes.
|
||||
|
||||
### Requirement 5: Reproduction and regression
|
||||
|
||||
**Objective:** As a maintainer, I want the bug-857 divergence reproduced and guarded, so that the fix is proven.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When the regression scenario interleaves a configuration upgrade with chaotic multi-kill failover, the test shall assert that every member's materialized state and active configuration are identical afterwards.
|
||||
2. The test shall reproduce the bug-857 materialized-state divergence on the pre-fix code (config-push + per-node replay) and pass on the fixed code.
|
||||
23
.kiro/specs/config-upgrade-snapshot/spec.json
Normal file
23
.kiro/specs/config-upgrade-snapshot/spec.json
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"feature_name": "config-upgrade-snapshot",
|
||||
"created_at": "2026-06-21T11:30:00Z",
|
||||
"updated_at": "2026-06-21T11:30:00Z",
|
||||
"language": "en",
|
||||
"phase": "requirements-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": ["entity-bearing-snapshots"]
|
||||
}
|
||||
|
|
@ -237,3 +237,77 @@ The only new data is the reply payload above. `CtrlStatus` is a flat value objec
|
|||
## Security Considerations
|
||||
|
||||
`ctrlstatus` exposes cluster topology, role, term, and config version/checksum — the same trust level as the existing `clusterconf`/state-report reads (no secrets, no config content, no job data). It is read-only and goes through the same authenticated client path as other client ops; no new authorization surface is introduced. Broadening the dispatch gate is scoped strictly to this one read opcode.
|
||||
|
||||
---
|
||||
|
||||
## Design Delta — Requirements 10 & 11 (2026-06-20)
|
||||
|
||||
> **Merge note (as-built correction).** The handler shipped as `ctrlstatus_fill` + `handle_ctrlstatus` in **`src/controller/ctrlstatus.c`** (not `reply.c`); `configChecksum` is carried as **Text** lowercase-hex (the `Data` capnp type maps to the project's structured `xce_data_t`); `primaryName` is **role-derived** (primary → self, else empty) after E2E fix `ade0081` — `ha->primary_host` is vestigial. The Req 10/11 changes are written against this as-built reality; the Reqs 1–9 design above stands as the original record.
|
||||
|
||||
### What changed and why
|
||||
Live E2E (2026-06-20): a standby's `ctrlstatus` reported `committedLsn: 0` / `committedConfigVersion: 0` while `appliedLsn: 38` — an apparent commit/apply inversion — because `ctrlstatus_fill` reads the **primary-only** markers `persist.committed_lsn` (`ctrlstatus.c:130`) and `ha->committed_config_tag` (`:109`), which stay `0` on a node that never led. And there was no responder timestamp to line a captured status up against `ctrl-N.log`. Req 10 fixes the off-primary committed sourcing; Req 11 adds a responder snapshot timestamp.
|
||||
|
||||
### Boundary Commitments (delta)
|
||||
- **This spec owns (added):** the off-primary field-sourcing fix inside `ctrlstatus_fill` (read the *learned* committed values when not primary); a new read-only `CtrlStatus.responderTimeMs` field stamped at snapshot time; its render in `cmd_ctrlstatus`.
|
||||
- **Out of Boundary (firm):** making the off-primary committed value **live-fresh** — i.e. consuming `HeartbeatMsg.committedLsn` / extending `OP_nop` to refresh `last_known_committed_lsn` on every heartbeat — is a heartbeat/replication-protocol change. It violates Req 10.4 and is routed to **ha-heartbeat-liveness** / **ha-repl-hardening**. controller-status reports the honest best-known value only.
|
||||
- **Allowed Dependencies (added):** read access to `ha->last_known_committed_lsn` (`_Atomic uint64_t`, `ha.h:170`) and `ha->last_known_committed_config_tag` (`uint64_t`, `ha.h:172`); `clock_gettime(CLOCK_REALTIME)`; the existing `xcelog.h` wall-clock format for client rendering.
|
||||
- **Revalidation Triggers (added):** any change to where/when `last_known_committed_lsn`/`last_known_committed_config_tag` are written (today: `cluster_apply_ctrlhello_reply` at join/RDHUP, and the election safe-commit path) → re-check the off-primary committed reporting and the staleness caveat.
|
||||
|
||||
### Locked design decisions (from research items)
|
||||
1. **Req 10 sourcing → role-branch (not uniform `max()`).** When `responderRole == CTRL_PRIMARY`, keep the node's own authoritative live markers (`persist.committed_lsn`, `ha->committed_config_tag`). Otherwise report the learned pair (`ha->last_known_committed_lsn`, `ha->last_known_committed_config_tag`). Rationale: a standby's learned value is its only honest "committed" knowledge; a primary's own marker is authoritative and must not be shadowed by a possibly-stale learned value. (Mirrors the `last_known_*` read precedent at `ha.c:2570`, `ha.c:4126`.)
|
||||
2. **Req 10 staleness caveat (documented behavior, not a bug).** `last_known_committed_lsn` is refreshed at **CtrlHello join / RDHUP-failover**, not per heartbeat, so an off-primary committed value MAY lag the live primary between joins. This is the in-boundary contract; live-fresh is the out-of-boundary item above. The `0`-inversion is eliminated regardless (a standby that has joined has a non-zero learned value; a never-joined node correctly reports `0` = unknown, per Req 1.3 / 10.2).
|
||||
3. **Req 11 wire type → `UInt64` epoch-milliseconds.** Numeric, locale-safe; the client formats it. (Pre-formatted `Text` was the alternative; rejected to keep the wire value machine-comparable and avoid encoding a locale into the reply.)
|
||||
4. **Req 11 clock → `CLOCK_REALTIME`, stamped in `ctrlstatus_fill` at snapshot time.** NOT `CLOCK_MONOTONIC` (which `ha_on_heartbeat_received` uses for liveness) — monotonic is not wall-clock and cannot be correlated to logs. The client renders the millis to local `YYYY-MM-DD HH:MM:SS,mmm` using the `xcelog.h` convention so it is directly comparable to a log line.
|
||||
|
||||
### Research item carried to implementation
|
||||
- Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h` and reuse it (or match its `YYYY-MM-DD HH:MM:SS,mmm` form) in `cmd_ctrlstatus` so the printed timestamp is byte-comparable to `ctrl-N.log`. `last_known_committed_config_tag` is non-atomic (`ha.h:172`, written under lock at `ha_convergence.c:377`); the status read is a best-effort aligned 8-byte load (acceptable for a reporting snapshot) — do not add locking on the read path.
|
||||
|
||||
### Schema delta (Cap'n Proto)
|
||||
Append one field to `CtrlStatus` (next free ordinal `@13`, after `peerCount @12`; append-only, never reorder):
|
||||
```capnp
|
||||
responderTimeMs @13: UInt64 $C.mapname("responder_time_ms"); # CLOCK_REALTIME epoch millis at snapshot (Req 11.1)
|
||||
```
|
||||
No change to `committedLsn @8` / `committedConfigVersion @7` wire shape — Req 10 changes only the **value source**, not the field.
|
||||
|
||||
### `ctrlstatus_fill` delta — Service contract (`src/controller/ctrlstatus.c`)
|
||||
- **Req 10 (committed sourcing):** replace the unconditional reads at `ctrlstatus.c:108–110,130` with the role-branch above:
|
||||
- `responderRole == CTRL_PRIMARY` → `committedLsn = persist.committed_lsn`, `committedConfigVersion = ha->committed_config_tag` (atomic acquire loads, unchanged).
|
||||
- else → `committedLsn = ha->last_known_committed_lsn` (atomic acquire), `committedConfigVersion = ha->last_known_committed_config_tag` (plain aligned load).
|
||||
- `appliedLsn` (`ha->applied_lsn`) is unchanged — it is already correct in every role.
|
||||
- **Req 11 (timestamp):** stamp `st->responder_time_ms` from `clock_gettime(CLOCK_REALTIME)` (`tv_sec*1000 + tv_nsec/1e6`) at the top of the snapshot, before/with the other field loads, so it marks the responder's sample moment.
|
||||
- **Read-only invariant (10.4, 1.2) preserved:** all additions are atomic/aligned loads + a clock read; no HA/persist/config mutation.
|
||||
|
||||
### `cmd_ctrlstatus` delta — State contract (`src/controller/testsuite/test.c`)
|
||||
- Print one added line rendering `responder_time_ms` as local `YYYY-MM-DD HH:MM:SS,mmm` (xcelog.h form) alongside the existing fields (Req 11.3). The committed lines need no rendering change — Req 10 only changes the values they already print.
|
||||
|
||||
### File Structure Plan (delta)
|
||||
**Modified files:**
|
||||
- `src/schema/controller.capnp` — add `responderTimeMs @13: UInt64` to `CtrlStatus` (regen `controller.capnp.h/.c` → `xce_ctrlstatus_t.responder_time_ms`).
|
||||
- `src/controller/ctrlstatus.c` — `ctrlstatus_fill`: role-branch committed sourcing (Req 10) + `CLOCK_REALTIME` timestamp stamp (Req 11).
|
||||
- `src/controller/testsuite/test.c` — `cmd_ctrlstatus`: render the responder timestamp line (Req 11.3).
|
||||
- `src/controller/testsuite/test_ctrlstatus.c` — add/extend unit tests (below).
|
||||
|
||||
No new files (the capnp field is generated; all logic slots into existing TUs).
|
||||
|
||||
### Requirements Traceability (delta)
|
||||
| Requirement | Summary | Components | Interfaces |
|
||||
|-------------|---------|------------|------------|
|
||||
| 10.1 | Off-primary reports *learned* committed LSN + config version | `ctrlstatus_fill` role-branch | `CtrlStatus.committedLsn`, `committedConfigVersion` (value source = `ha->last_known_committed_*`) |
|
||||
| 10.2 | No learned value yet → 0/unknown | `ctrlstatus_fill` | `last_known_committed_lsn == 0` (correct unknown, per 1.3) |
|
||||
| 10.3 | No spurious applied>committed inversion | `ctrlstatus_fill` | `appliedLsn` vs role-correct `committedLsn` |
|
||||
| 10.4 | Read-only reporting; no production/replication change | `ctrlstatus_fill`; out-of-boundary routing | atomic/aligned loads only |
|
||||
| 11.1 | Responder wall-clock at sample time | `ctrlstatus_fill` (`CLOCK_REALTIME`) | `CtrlStatus.responderTimeMs` |
|
||||
| 11.2 | Same clock/format as logs | `cmd_ctrlstatus` render (xcelog.h form) | local `YYYY-MM-DD HH:MM:SS,mmm` |
|
||||
| 11.3 | Test client prints the timestamp | `cmd_ctrlstatus` | added print line |
|
||||
|
||||
### Testing Strategy (delta)
|
||||
**Unit (`test_ctrlstatus.c`, TDD red→green):**
|
||||
- STANDBY with `last_known_committed_lsn = 36`, `persist.committed_lsn = 0`, `applied_lsn = 38` → reply reports `committedLsn == 36` (not 0) and `appliedLsn == 38` (no inversion below the learned value) (10.1, 10.3).
|
||||
- STANDBY pre-join (`last_known_committed_lsn == 0`) → reply reports `committedLsn == 0` / `committedConfigVersion == 0` (correct unknown) (10.2).
|
||||
- PRIMARY → reply reports `committedLsn == persist.committed_lsn` and `committedConfigVersion == ha->committed_config_tag` (authoritative own markers unchanged) (10.1 primary branch).
|
||||
- `committedConfigVersion` off-primary equals `last_known_committed_config_tag` (symmetry with the LSN) (10.1).
|
||||
- `responder_time_ms` is non-zero and within a tolerance of a test-captured `CLOCK_REALTIME` sample; the handler still mutates no state (11.1, 1.2/10.4).
|
||||
|
||||
**Integration / round-trip (libxce):** `xcecli_ctrlstatus` decodes the new `responderTimeMs` field end-to-end (no manual decode — codegen), proving wire compatibility.
|
||||
|
||||
**E2E (live `/xce-deploy-ha`):** reproduce the original finding — submit jobs so the primary commits, query a **standby**; assert the standby's `committedLsn` is non-zero and tracks (within the join-refresh caveat) rather than `0`, and that the printed `responder timestamp` matches the standby's own `ctrl-N.log` line for that query.
|
||||
|
|
|
|||
|
|
@ -1,31 +1,117 @@
|
|||
# Requirements Document
|
||||
|
||||
## Project Description (Input)
|
||||
## Introduction
|
||||
|
||||
Add an LSF-`lsid`-style controller status opcode (C tier only) so a client can query basic cluster/controller identity and health in one cheap call.
|
||||
This feature adds a **controller status query** (an XCE analog of LSF's `lsid`): a single, cheap request that returns basic cluster/controller identity and health — which node is the current primary, the answering node's identity and role, the configured controllers (peers), the committed configuration version and a content checksum, and a few HA health/progress fields. Today no such query exists: a client must infer "is there a serving primary" indirectly (for example, by listing jobs), and there is no single call that reports who the primary is, the configured controller topology, or the committed configuration version. This makes operational visibility and failover-aware test orchestration awkward.
|
||||
|
||||
**Returns:**
|
||||
- Which node is the current PRIMARY (name; empty/none if mid-election)
|
||||
- The responding node's role (PRIMARY/STANDBY/ELECTING/INIT)
|
||||
- The configured controllers/peers (name + host:port list and count)
|
||||
- The configuration version (the HA committed `config_tag`)
|
||||
- Additional status: current term (election epoch), `committed_lsn`/`applied_lsn` (catch-up/health), cluster name, serving/fenced state
|
||||
The status query is served by **any** controller node — a standby reports who the primary is from the state it already tracks — and rides the existing client failover behavior, so a caller can get status without knowing which node is primary. The test client also gains a command that prints this status, and its `wait-primary` probe switches to this query so that waiting for a primary is cheaper and additionally reports which node became the serving primary.
|
||||
|
||||
Answerable by ANY node (a standby reports who the primary is from its ctrlhello/heartbeat state), so `lsid` works even when the query doesn't land on the primary; it is a failover-aware READ like the existing `clusterconf` op. All the underlying data already exists in `src/controller/ha.h` (`ha->role`, `ha->term`, `committed_config_tag`, `committed_lsn`, peer/candidate list).
|
||||
## Boundary Context
|
||||
|
||||
**Components (C tier):**
|
||||
1. `src/schema/controller.capnp` — new `CtrlInfo`/`CtrlStatus` struct + request opcode + reply-union member, modeled on the existing `clusterconf` opcode (`controller.capnp:489`).
|
||||
2. Controller `handle_ctrlinfo` handler mirroring `handle_clusterconf`.
|
||||
3. libxce `xce_ctrlinfo_req` / `xce_ctrlinfo` / `xcecli_ctrlinfo` (+ free helper) as a failover-aware read.
|
||||
4. `src/controller/testsuite/test.c` — a `ctrlinfo`/`lsid` client command that prints the status, AND switch `cmd_wait_primary`'s probe from `joblist_query` to this lighter, more-direct status op. (`wait-primary` currently piggybacks on `joblist` purely to avoid a new opcode; with this op it becomes a cheap "is a serving primary present + who" probe that can also report the elected primary and term — better diagnostics, directly useful for the `ha-commit-durability` Phase 6 liveness / >60s-election investigation.)
|
||||
|
||||
**KEY DESIGN DECISION the spec must resolve:** whether to EXTEND the existing `clusterconf` opcode (which already returns much richer cluster detail — name/resources/queues/workers) with these HA-status fields, OR add a separate lean status op. `lsid` wants a fast focused identity/health view, not the full resource/queue/worker dump, so the leaning is a dedicated lean op, but the spec should weigh reuse vs separation explicitly.
|
||||
|
||||
**IN SCOPE:** C tier only (controller opcode + libxce + `xce_test` command + `wait-primary` integration).
|
||||
|
||||
**OUT OF SCOPE:** the Go CLI/REST `xce lsid` surface (a later pass); changing HA/election/replication behavior (this only READS existing HA state); structured/JSON output beyond the capnp reply.
|
||||
|
||||
This is the "controller status/probe opcode" that the `xce-test-e2e-primitives` spec explicitly deferred as out-of-scope.
|
||||
- **In scope**: a client-observable controller status query returning the current primary identity, the answering node's identity and role, the configured controller topology (peers + count), the committed configuration version and content checksum, and HA health (current term, commit/apply progress, serving/fenced state, cluster name); off-primary fidelity of the reported committed positions (a standby reports the committed position it has learned from the primary, not an unset node-local value); a responder snapshot timestamp for log correlation; failover-aware delivery of that query; a test-client command that prints the status; and `wait-primary` using the status query as its probe.
|
||||
- **Out of scope**: an operator-facing CLI/REST status surface (a later pass); any change to HA election, replication, or commit behavior (this query only **reads** existing controller state); presentation/output formats beyond the query's structured reply and the test client's human-readable print.
|
||||
- **Adjacent expectations**: relies on the existing client master-rotation/failover already used by other read operations; consumes the controller's existing HA state (current primary, role, term, committed configuration version, commit/apply progress, fenced state) without changing how that state is produced or maintained. The status query is a **focused identity/health view** and is intentionally distinct from the existing full cluster-configuration listing (which enumerates resources, queues, and workers).
|
||||
|
||||
## Requirements
|
||||
<!-- Will be generated in /kiro-spec-requirements phase -->
|
||||
|
||||
### Requirement 1: Single-call controller status query
|
||||
|
||||
**Objective:** As an operator or test author, I want to retrieve basic controller and cluster status in a single request, so that I can see cluster identity and health without inferring it from unrelated operations.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a controller status query, the Controller shall return, in a single request/response exchange, a response containing the status fields defined in Requirements 2 through 6.
|
||||
2. The Controller shall treat the status query as read-only and shall not modify any cluster or controller state as a result of answering it.
|
||||
3. If the responding controller cannot determine the value of a status field, then the Controller shall return a well-defined "unknown" value for that field rather than omitting it or returning uninitialized data.
|
||||
4. The status response shall be limited to controller/cluster identity and health fields and shall not include the full resource, queue, or worker inventory.
|
||||
|
||||
### Requirement 2: Current primary identity
|
||||
|
||||
**Objective:** As an operator, I want the status response to identify the current primary, so that I know which node is authoritative.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. While a primary is currently elected and serving, when a client issues a status query, the Controller shall report the current primary's identity in the response.
|
||||
2. If no primary has currently been elected, then the Controller shall report the primary identity as "unknown" rather than a stale or guessed value.
|
||||
3. When the status query is answered by a standby node that knows the current primary, the Controller shall report that primary's identity; if the standby does not currently know a primary, the Controller shall report the primary identity as "unknown".
|
||||
|
||||
### Requirement 3: Responding node identity and role
|
||||
|
||||
**Objective:** As an operator, I want to know which node answered and its role, so that I can interpret the response in context without confusing it with the authoritative cluster view.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall report the identity (name) of the node that answered the query.
|
||||
2. When a client issues a status query, the Controller shall report the answering node's current role as one of: primary, standby, electing, or initializing.
|
||||
3. The responding node's role shall be presented as a property of the answering node (for example, "answered by <node> (<role>)") and shall be distinct from the authoritative current-primary identity of Requirement 2, so that a transient answering-node role (such as electing or initializing) is not misread as the cluster's primary state.
|
||||
|
||||
### Requirement 4: Configured controller topology
|
||||
|
||||
**Objective:** As an operator, I want the configured controllers, so that I can see the cluster's intended membership and size.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall report the list of configured controllers, each identified by its name and network address.
|
||||
2. When a client issues a status query, the Controller shall report the total count of configured controllers.
|
||||
|
||||
### Requirement 5: Committed configuration version and checksum
|
||||
|
||||
**Objective:** As an operator, I want the configuration version and a content checksum, so that I can confirm which configuration the cluster has committed and detect content divergence between nodes even when their version numbers match.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall report the committed configuration version currently in effect on the responding node.
|
||||
2. When a client issues a status query, the Controller shall report a checksum (CRC) of the committed configuration content currently in effect on the responding node.
|
||||
3. The reported configuration checksum shall be derived from the configuration content such that two nodes with identical committed configuration content report the same checksum, and two nodes with differing content report different checksums (enabling detection of same-version content divergence).
|
||||
|
||||
### Requirement 6: HA health and progress fields
|
||||
|
||||
**Objective:** As an operator, I want basic HA health and progress, so that I can assess election state and catch-up.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall report the current term (election epoch).
|
||||
2. When a client issues a status query, the Controller shall report its commit progress position and its apply progress position.
|
||||
3. While the responding node is in a fenced or non-promotable state, when a client issues a status query, the Controller shall report that fenced state in the response.
|
||||
4. When a client issues a status query, the Controller shall report the cluster name.
|
||||
|
||||
### Requirement 7: Failover-aware, any-node delivery
|
||||
|
||||
**Objective:** As a client, I want the status query to reach a reachable controller automatically, so that I can get status even across failover or when I do not know which node is primary.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the XCE client shall attempt the configured controllers using the same failover/rotation behavior used by other read operations until one responds or the attempts are exhausted.
|
||||
2. When any configured controller, whether primary or standby, is reachable, the XCE client shall return that controller's status response.
|
||||
3. If no configured controller is reachable within the client's failover budget, then the XCE client shall report a typed failure rather than reporting success with empty status.
|
||||
|
||||
### Requirement 8: Test-client status command
|
||||
|
||||
**Objective:** As a test author, I want a client command that prints the controller status, so that I can inspect cluster identity and health from a script.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When the status command is issued in the test client, the test client shall print, in human-readable form, the current primary identity, the answering node's identity and role (as "answered by"), the configured controllers and their count, the committed configuration version, the committed configuration checksum, the current term, the commit/apply progress, the fenced state, and the cluster name.
|
||||
2. When the underlying status query succeeds, the test client command shall report success (zero status).
|
||||
3. If the underlying status query fails, then the test client command shall report failure (non-zero status) and shall print the error.
|
||||
|
||||
### Requirement 9: wait-primary uses the status query
|
||||
|
||||
**Objective:** As a test author, I want `wait-primary` to probe via the status query, so that the wait is cheaper and reports which node became the serving primary.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When `wait-primary` probes the cluster, the wait-primary command shall use the controller status query, rather than a job-listing operation, to determine whether a serving primary is present.
|
||||
2. When the status query reports a serving primary within the timeout, the wait-primary command shall complete successfully and shall report the identity of the serving primary.
|
||||
3. If no serving primary is reported within the timeout, then the wait-primary command shall fail.
|
||||
4. The change to the wait-primary probe shall preserve its existing timeout-argument handling and its success/failure exit semantics, adding only the served-primary identity to its success output.
|
||||
|
||||
### Requirement 10: Off-primary committed-state fidelity
|
||||
|
||||
**Objective:** As an operator, I want a standby's reported committed positions to reflect what it has actually learned from the primary, so that the status does not show a misleading committed value of zero while the node has clearly applied replicated entries.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. While the responding node is not the current primary and has learned a committed position from the primary, when a client issues a status query, the Controller shall report the committed configuration version and commit progress position that it has learned from the primary, rather than a node-local committed value that remains unset on a node that has never served as primary.
|
||||
2. If the responding node is not the current primary and has not yet learned any committed position from a primary, then the Controller shall report the committed configuration version and commit progress position as the well-defined "unknown"/zero value (consistent with Requirement 1.3), rather than a stale or guessed value.
|
||||
3. While the responding node is actively applying replicated entries, the Controller shall source the reported commit progress position and apply progress position consistently, so that the reported commit position is not spuriously lower than the position the node has actually committed or learned — avoiding an apparent "applied greater than committed" inversion that does not reflect the node's real state.
|
||||
4. The committed-state fidelity in this requirement shall be a read-only reporting concern only and shall not change how committed positions are produced, replicated, or persisted.
|
||||
|
||||
### Requirement 11: Responder snapshot timestamp for log correlation
|
||||
|
||||
**Objective:** As an operator or test author, I want each status response to carry the responding node's clock time at the moment it sampled its state, so that I can line a captured status up against that node's logs.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a client issues a status query, the Controller shall include in the response the responding node's wall-clock time at the moment the status fields were sampled.
|
||||
2. The reported timestamp shall be expressed on the same clock and in a form that an operator can directly compare against the responding controller's log timestamps.
|
||||
3. When the status command is issued in the test client, the test client shall print the responder snapshot timestamp in human-readable form alongside the other status fields.
|
||||
|
|
|
|||
|
|
@ -91,3 +91,63 @@ Two review issues resolved with the user; design.md updated:
|
|||
2. **Config fields → loaded pair + committed tag (user decision).** Report `configVersion`+`configChecksum` from the LOADED config (self-consistent: version = first 8 bytes of the checksum) AND `committedConfigVersion` (`committed_config_tag`) separately — "what I run" vs "what is committed". Avoids the earlier mismatched committed-tag + loaded-checksum pairing.
|
||||
|
||||
Also fixed: mermaid render issues — removed `/`, parentheses, and quotes from diagram labels/messages (the LR architecture graph + the new redirect sequence diagram).
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis — Delta for Requirements 10 & 11 (2026-06-20)
|
||||
|
||||
Reopened after a live HA-deploy E2E finding: a standby answering `ctrlstatus` showed
|
||||
`committedLsn: 0` / `committedConfigVersion: 0` while `appliedLsn: 38` (an apparent
|
||||
applied-greater-than-committed inversion), and there was no way to correlate a captured
|
||||
status against the per-node logs. Scope of this delta = Req 10 (off-primary committed
|
||||
fidelity) + Req 11 (responder snapshot timestamp). Reqs 1–9 are implemented and unchanged.
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
| Need (Req) | Existing asset | Gap |
|
||||
|---|---|---|
|
||||
| 10.1 standby reports *learned* committed LSN | `ctrlstatus_fill` reads `c->persist.committed_lsn` (`ctrlstatus.c:130`) — a **primary-only** marker, 0 on a node that never led | **Constraint**: wrong source field; the learned value `ha->last_known_committed_lsn` (`ha.h:170`) exists but is unused here |
|
||||
| 10.1 standby reports *learned* committed config tag | `ctrlstatus_fill` reads `ha->committed_config_tag` (`ctrlstatus.c:109`) — also primary-only | **Constraint**: learned `ha->last_known_committed_config_tag` (`ha.h:172`) exists but is unused here |
|
||||
| 10.1 freshness of the learned value | `last_known_committed_lsn` written only at `cluster_apply_ctrlhello_reply` (`cluster.c:291`, `ha_convergence.c:375`) and an election safe-commit path (`ha.c:925`) | **Constraint (key)**: refreshed only at **CtrlHello join / RDHUP-failover handshake**, NOT per heartbeat. The `OP_nop` heartbeat (`cluster.c:300`) carries no committed payload; the richer `HeartbeatMsg{committedLsn@2, committedConfigTag@3}` (`controller.capnp:761`) is **defined but never consumed** on receive → the learned value can LAG between joins |
|
||||
| 10.2 genuinely-unknown → 0/unknown | existing `NULLSTR`/0 conventions (Req 1.3) | none — `last_known_committed_lsn` is 0 before first learn, which is the correct "unknown" |
|
||||
| 11.1 responder wall-clock at sample time | none on `CtrlStatus` (schema `controller.capnp:496`, fields @0–@12; next free ordinal **@13**) | **Missing**: new field + a `CLOCK_REALTIME` stamp in `ctrlstatus_fill`. Note `ha_on_heartbeat_received` uses `CLOCK_MONOTONIC` — **not** reusable for log correlation |
|
||||
| 11.2 same clock/format as logs | logger header `src/include/xcelog.h` produces `[YYYY-MM-DD HH:MM:SS,mmm]` | **Research Needed**: confirm the exact formatter/helper in `xcelog.h` to reuse (or match) for byte-comparable output |
|
||||
| 11.3 test client prints timestamp | `cmd_ctrlstatus` in `src/controller/testsuite/test.c` (C `xce_test` client; print block ~line 1297) — the "client" is xce_test, not Go | **Missing**: one print line; libxce decode is codegen from schema (auto after regen) |
|
||||
|
||||
### Implementation Approach Options
|
||||
|
||||
**Req 10 — Option A (Extend `ctrlstatus_fill`) — RECOMMENDED.** When `responderRole != CTRL_PRIMARY`,
|
||||
source `committedLsn` from `ha->last_known_committed_lsn` and `committedConfigVersion` from
|
||||
`ha->last_known_committed_config_tag`; keep the primary's own `persist.committed_lsn` /
|
||||
`committed_config_tag` when it IS primary (those are the authoritative live values). Read-only,
|
||||
satisfies Req 10.4, mirrors the existing `last_known_*` usage in the RDHUP/freshness paths
|
||||
(`ha.c:2570`, `ha.c:4126`). A simpler variant — report `max(persist.committed_lsn,
|
||||
last_known_committed_lsn)` uniformly (no role branch) — also kills the 0-inversion; design picks.
|
||||
*Trade-off*: ✅ tiny, in-boundary, fixes the inversion. ❌ value can lag between joins (see below).
|
||||
|
||||
**Req 10 — out-of-boundary (do NOT pull into this spec).** Making the off-primary committed
|
||||
value *live-fresh* requires consuming `HeartbeatMsg.committedLsn` (or extending `OP_nop`) and
|
||||
storing it into `last_known_committed_lsn` on **every** heartbeat — a heartbeat/replication
|
||||
protocol change. That violates Req 10.4 ("shall not change how committed positions are produced,
|
||||
replicated, or persisted") and belongs to **ha-heartbeat-liveness** / **ha-repl-hardening**.
|
||||
controller-status should report the honest best-known value and the design must state the lag caveat.
|
||||
|
||||
**Req 11 — Option A (Extend) — RECOMMENDED.** Add `responderTime @13: UInt64` (epoch millis) to
|
||||
`CtrlStatus`; stamp it in `ctrlstatus_fill` from `CLOCK_REALTIME`; render in `cmd_ctrlstatus`
|
||||
formatted via the `xcelog.h` convention so it lines up with `ctrl-N.log`. *Design decision*:
|
||||
numeric epoch-millis (client formats — locale-safe, clean) vs pre-formatted `Text` (zero client
|
||||
work, mirrors the `configChecksum`-as-Text precedent). Recommend numeric epoch-millis.
|
||||
*Trade-off*: ✅ one field + one stamp + one print line. ❌ adds a schema ordinal (codegen regen).
|
||||
|
||||
### Effort & Risk
|
||||
|
||||
- **Req 10**: **S** (≤1 day) — one function's field sourcing + TDD (standby-known-primary asserts learned value; pre-learn asserts 0). **Risk Low** — read-only, no HA-state change.
|
||||
- **Req 11**: **S** (1–2 days) — schema field + `ctrlstatus_fill` stamp + `cmd_ctrlstatus` print + codegen regen + round-trip decode test. **Risk Low**.
|
||||
- **Combined**: **S**. The one **Medium-risk temptation to refuse**: expanding into heartbeat committed-propagation — keep it out of this spec.
|
||||
|
||||
### Research Items to Carry to Design
|
||||
|
||||
1. Confirm the exact wall-clock formatter/helper in `src/include/xcelog.h`; reuse it (or document the chosen format) so Req 11's value is byte-comparable to log lines.
|
||||
2. Decide Req 11 wire type: `UInt64` epoch-millis (recommended) vs `Text`.
|
||||
3. Decide Req 10 sourcing: role-branch (`persist.*` on primary, `last_known_*` off-primary) vs uniform `max()`; keep LSN and config-tag symmetric.
|
||||
4. State the **staleness caveat** in design (learned value refreshes at join, not per heartbeat) and explicitly route live-fresh-committed to **ha-heartbeat-liveness / ha-repl-hardening** as out-of-boundary.
|
||||
|
|
|
|||
|
|
@ -379,3 +379,140 @@ Since the spec is implemented, options address only the remaining gap closure.
|
|||
- **Duplicate-vote semantics decision**: confirm against `raft/raft.tla` (Election Safety invariant) that same-candidate re-grant is safe in XCE's persistence model (vote persisted before grant → re-grant is replay-safe). Expected answer: yes; then align ha.c to raft.c.
|
||||
- **"self" sentinel scenario**: reproduce with deploy-ha — node self-votes (candidate), is killed mid-election, restarts within the same term; verify it does not skip its own election indefinitely.
|
||||
- **Full regression + ASAN pass**: run `/xce-build controller` after fixes; this audit ran only `controller_unit_test` (2 failures, no ASAN errors observed in that run).
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
> Post-implementation audit. Spec phase: `implemented` (all tasks `[x]`). This pass
|
||||
> re-verifies the 9 requirements against the CURRENT on-disk codebase, which has
|
||||
> evolved substantially since the 2026-06-09 pass via four downstream specs that
|
||||
> landed on this branch: `ha-single-leader-role`, `ha-prevote`,
|
||||
> `ha-convergence-recovery`, and `ha-commit-durability`.
|
||||
> Evidence gathered via codegraph + grep (no build run this pass).
|
||||
|
||||
### Headline finding
|
||||
|
||||
The async iomgr election mechanism this spec introduced is **intact and in
|
||||
production** (Req 1–3, 6, 7, 9). However, the spec's **COMMITTING availability-gate
|
||||
model (Req 4.1, 5.2, 8.1–8.4) was deliberately superseded** by `ha-single-leader-role`:
|
||||
|
||||
- `CTRL_PRIMARY_COMMITTING` enum value **deleted** (commit 93dfc02). The enum in
|
||||
`ha.h:60–64` now has only `CTRL_PRIMARY / CTRL_STANDBY / CTRL_INIT / CTRL_ELECTING`.
|
||||
- The dispatch-gate COMMITTING branch **deleted** (71ce41f); `ha_dispatch_gate`
|
||||
(ha.c:2061) now has 4 role cases, no COMMITTING (ha.c:2067/2110/2127/2145).
|
||||
- The force-promotion watchdog + ack/convergence machinery **deleted** (a5bf861):
|
||||
`ha_repl_ack_received`, `ha_check_majority_convergence`, `committing_timeout_fn`
|
||||
are all absent from the tree (grep: 0 matches).
|
||||
- Promotion now goes **direct to CTRL_PRIMARY** via a shared sequence
|
||||
(`ha_promote_to_primary` → hooks + heartbeat + **own-term WAL_NOOP + writable_lsn
|
||||
gate**, ha.c:2298). The durability guarantee the COMMITTING gate was meant to
|
||||
provide is now enforced by the own-term commit gate (`ha_arm_own_term_gate`,
|
||||
ha.c:2259) and the durable-backed vote freshness floor — both owned by
|
||||
`ha-commit-durability`.
|
||||
- Test artifact `test_ha_evaluate_committing.c` (cited in earlier audits) is **gone**.
|
||||
|
||||
This is **accepted, intentional drift**, not a regression: a later approved spec
|
||||
replaced a weaker availability gate with a stronger durability invariant. The
|
||||
requirement *intent* (a promoted primary holds current state and never serves stale
|
||||
data) is **better** satisfied now. The requirement *wording* (specific
|
||||
`CTRL_PRIMARY_COMMITTING` state + majority-ack counter) is obsolete.
|
||||
|
||||
### Status of the two failures flagged on 2026-06-09
|
||||
|
||||
Both prior-audit failures appear resolved in the current tree (Option A reconciliation
|
||||
was applied):
|
||||
- **`primary_ctrl_fd >= 0` deny gate** is now intentional production code
|
||||
(`handle_request_vote`, ha.c:320: `role == CTRL_STANDBY && ha->primary_ctrl_fd >= 0
|
||||
&& ha_heartbeat_is_fresh(ha)`), with an explicit comment block (ha.c:312–319)
|
||||
justifying it against the RDHUP / liveness-watchdog case. Matches Req 6.2's "live
|
||||
primary **connection**" wording.
|
||||
- **`myapp_run` "self" sentinel** now also compares the node's real name
|
||||
(myapp.c:281–283: `voted_for_name != "" && != "self" && != myconf->name`),
|
||||
closing the self-vote-skips-own-election hole the prior audit raised.
|
||||
> Caveat: not re-run this pass (no build). If closing the branch, run
|
||||
> `/xce-build controller` to confirm `controller_unit_test` is green.
|
||||
|
||||
### Requirement-to-Asset Map (current code, tagged)
|
||||
|
||||
| Req | Summary | Current Asset (file:line) | Tag |
|
||||
|-----|---------|---------------------------|-----|
|
||||
| 1.1–1.3 | No blocking vote pthreads; vote via standard I/O; probe timeout = deny | `ha_start_election_async` (ha.c:4400) → `raft_start_election` (raft.c:61); timeout via `raft_election_timeout_fn` (ha.c:342) + `delay_task_ms`; `vote_request_thread`/`am_i_primary_quorum` removed | MET |
|
||||
| 2.1–2.4 | Async TCP+TLS via iomgr; TLS client ctx; vote after handshake; connect fail = deny | `endpoint_set_on_connected` (netio.c) fire-once; `on_vote_connected` → `xce_request`; `client_ssl_ctx` in `ha_state_t`; `vote_ep_err_handler` (ha.c:3937) counts denied vote | MET |
|
||||
| 3.1–3.4 | Reply via iomgr dispatch; no capnp UAF; field extraction; term fencing | `peer_ctrl_msg_handler` (ha.c:3600) role-routes; `handle_request_vote_reply` (ha.c:3017) reads fields before free; term fencing in `raft_on_vote_reply` (raft.c:338) | MET |
|
||||
| 4.1 | Quorum → CTRL_PRIMARY_COMMITTING (pre-availability) | **SUPERSEDED**: quorum → direct `CTRL_PRIMARY` via `raft_cb_become_leader` / `ha_promote_to_primary` (ha.c:2298); COMMITTING state deleted | Constraint (intentional drift — ha-single-leader-role) |
|
||||
| 4.2–4.3 | Freshness gate = ground truth; cold-start only highest config_tag wins | `raft_cb_vote_eligible` → `ha_vote_self_freshness` (ha.c:2550) per-field max(local, learned-committed); durable-backed floor via `ha_candidate_advertise_lsn`/`persist_durable_floor` (ha.c:2587) [ha-commit-durability 4.2]; config-less node refuses to campaign | MET (strengthened) |
|
||||
| 4.4–4.5, 4.7 | Split-vote / no-reply retry; retry indefinitely | `raft_on_election_timeout` (raft.c:399), `raft_on_leader_lost` (raft.c:417) priority-scaled backoff; heartbeat watchdog re-arm | MET |
|
||||
| 4.6 | ELECTING + heartbeat from new PRIMARY → STANDBY | Equivalent mechanism (vote-grant → `raft_cb_become_follower`; winner ctrlhello retried), not literal heartbeat→STANDBY. Documented accepted drift since 2026-06-09 | MET (mechanism drift, documented) |
|
||||
| 5.1 | ELECTING gate: requestVote/probeConfigTag only, else NOT_READY | `ha_dispatch_gate` CTRL_ELECTING case (ha.c:2127); PreVote opcodes added by ha-prevote | MET |
|
||||
| 5.2 | COMMITTING: repl acks only | **SUPERSEDED**: no COMMITTING state/gate; promotion is atomic to PRIMARY. Durability via own-term gate instead | Constraint (intentional drift) |
|
||||
| 5.3–5.4 | PRIMARY full traffic; STANDBY repl/heartbeat | `ha_dispatch_gate` CTRL_PRIMARY (ha.c:2067) / CTRL_STANDBY (ha.c:2145) | MET |
|
||||
| 5.5 | Monotonic persisted term across restart | `ha_persist_term`/`ha_restore_term`; `raft_cb_persist` persists before grant; `ha_init_raft` (ha.c:3991) seeds raft from restored term **once** at startup (myapp.c:242) | MET |
|
||||
| 6.1 | PRIMARY denies all votes | raft.c role gate + legacy `handle_request_vote` gate (COMMITTING clause now dead/removed — only PRIMARY remains) | MET |
|
||||
| 6.2 | STANDBY + live primary denies votes | `handle_request_vote` (ha.c:320): `CTRL_STANDBY && primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh`. Prior failing test reconciled | MET |
|
||||
| 6.3, 6.5 | HB timeout → ELECTING → new primary | heartbeat watchdog → `ha_on_primary_ctrl_fd_rdhup` → `ha_start_election_async`; `test_ha_rdhup_integration.c` registered (meson.build:123) | MET |
|
||||
| 6.4 | INIT/ELECTING evaluate term/voted_for/freshness | `raft_on_vote_request` (raft.c:110); dual-path divergence flagged 2026-06-09 — reconcile/retire still tracked (see residual gaps) | MET (with residual dual-path debt) |
|
||||
| 7.1–7.3 | Winner pushes config + WAL; tag=0 standby never fetches | `push_config_to_standbys` + `ha_send_ctrlhello_to_all_standbys` (cluster.c:1629); heartbeat-driven repl decision loop; `fetch_config_from_primary` removed | MET |
|
||||
| 7.4 | Per-standby ack before counting toward majority | Re-homed: durable-commit majority via commit engine (`commit_on_ack`/`commit_advance_locked`, commit.c) + `standby_converged[]`; the spec's `repl_ack_count` counter deleted | MET (re-homed to commit engine) |
|
||||
| 8.1–8.3 | COMMITTING blocks jobs; majority ack → PRIMARY; PRIMARY opens traffic | **SUPERSEDED**: no COMMITTING. Job traffic gated by `ha_dispatch_gate` PRIMARY + own-term `writable_lsn` gate (commit.c:383) — WAIT completion withheld until `committed_lsn >= writable_lsn`. Stronger than the original ack-count gate | Constraint (intentional drift) |
|
||||
| 8.4 | Configurable ack timeout → degraded open | **SUPERSEDED**: committing timeout deleted; degraded-mode handled by ha-commit-durability's non-promotable fence + bounded catch-up (commit 59d9df2/addf53e) | Constraint (intentional drift) |
|
||||
| 9.1 | Single-node self-elect, no peer contact | `raft_start_election` n_peers==1 shortcircuit (raft.c:87) | MET |
|
||||
| 9.2 | Winner never fetches config | Freshness gate + push model; no fetch call sites | MET |
|
||||
| 9.3 | RDHUP → re-election; promotion hooks-only | `ha_on_primary_ctrl_fd_rdhup` + `phase2_done` skip; `ha_promote_to_primary` shared sequence | MET |
|
||||
| 9.4 | Priority stagger | `election_priority_step_ms` stagger (myapp.c); raft `my_priority` backoff (raft.c:421) | MET |
|
||||
|
||||
Tally: 9/9 requirements satisfied in **intent**. 6 of 9 requirement groups fully MET
|
||||
as written; 3 groups (4.1, 5.2, 8.1–8.4) carry **Constraint** tags — their literal
|
||||
wording is obsolete because the COMMITTING availability gate was intentionally
|
||||
replaced by a stronger durability gate in a later approved spec.
|
||||
|
||||
### Residual / deferred gaps
|
||||
|
||||
| # | Gap | Type | Notes |
|
||||
|---|-----|------|-------|
|
||||
| R1 | Requirements text still describes `CTRL_PRIMARY_COMMITTING` + majority-ack gate (Req 4.1, 5.2, 8.1–8.4) that no longer exist | Doc drift | requirements.md/design.md are now historically inaccurate vs. code. Spec is closed; this is documentation debt, not a code gap. |
|
||||
| R2 | Dual vote-path divergence (`handle_request_vote` in ha.c vs `raft_on_vote_request` in raft.c) | Constraint / tech debt | Carried from 2026-06-09. Production vote path is raft.c (via cluster.c); ha.c legacy path retained for tests. Option B (retire legacy path) still open. |
|
||||
| R3 | `ha_init_raft` one-time-init invariant has no dedicated unit test | Missing (test) | grep: no testsuite file drives `ha_init_raft` across two election cycles. Double-init would reset term/voted_for. |
|
||||
| R4 | Known dead code still compiled: `ha_wait_for_heartbeat_or_timeout`, `am_i_primary`, heartbeat-listener stubs; stale comments referencing deleted `fetch_config_from_primary` (ha.c) | Constraint (cosmetic) | Carried across every prior audit; no correctness impact. |
|
||||
| R5 | Build/ASAN not re-run this pass | Unknown | Audit was code-graph only. Confirm `controller_unit_test` green before relying on R1/the 2 reconciled tests. |
|
||||
|
||||
### Open-work options (only R1–R3 warrant action; R4 cosmetic, R5 verification)
|
||||
|
||||
**R1 — reconcile requirements text with superseded gate**
|
||||
- Option A: Add a "Superseded by ha-single-leader-role / ha-commit-durability" banner
|
||||
to requirements.md §4/§5/§8 (Effort S, Risk Low). *Recommended.*
|
||||
- Option B: Rewrite Req 4/5/8 to describe the own-term gate model (Effort M, Risk Low)
|
||||
— higher fidelity but rewrites an already-closed spec.
|
||||
- Option C: Leave as-is, rely on this research.md note (Effort 0, Risk Low) — readers
|
||||
of requirements.md alone stay misled.
|
||||
|
||||
**R2 — single vote-path consolidation** (from 2026-06-09 Option B)
|
||||
- Option A: Keep dual path, document raft.c as authoritative (Effort S, Risk Low).
|
||||
- Option B: Delete `handle_request_vote` from ha.c, re-plumb its tests onto
|
||||
`raft_on_vote_request` (Effort M, Risk Medium — election-safety semantics; verify
|
||||
against raft.tla Election Safety).
|
||||
- Option C: Hybrid — A now, B as a tracked cleanup-spec item. *Recommended.*
|
||||
|
||||
**R3 — `ha_init_raft` invariant test**
|
||||
- Option A: Add a unit test (init once, run two election cycles, assert term monotonic
|
||||
+ voted_for survives) using existing Unity patterns in test_ha_election.c
|
||||
(Effort S, Risk Low). *Recommended.*
|
||||
|
||||
| Open item | Effort | Risk |
|
||||
|-----------|--------|------|
|
||||
| R1 doc banner (Option A) | S | Low |
|
||||
| R2 vote-path consolidation (Option C) | S now / M later | Medium (semantics) |
|
||||
| R3 ha_init_raft test | S | Low |
|
||||
| R4 dead-code/stale-comment cleanup | S | Low |
|
||||
| R5 build + ASAN verification | S | Low |
|
||||
|
||||
### Conclusion
|
||||
|
||||
`election-iomgr-refactor` is **CLOSED** as an implementation: the async iomgr
|
||||
election it delivered is live and all 9 requirements are satisfied in intent. The
|
||||
only true open items are documentation/test debt (R1, R3) and pre-existing tech debt
|
||||
(R2, R4) — none block the feature. The single material caveat is that the
|
||||
availability-gate requirements (4.1/5.2/8.x) are now historically inaccurate because
|
||||
a later approved spec replaced the COMMITTING gate with a stronger durability gate;
|
||||
this should be recorded in requirements.md (R1) so the spec docs stop describing
|
||||
removed code.
|
||||
|
|
|
|||
|
|
@ -106,3 +106,65 @@ Upstream interface consumed (ha-commit-engine design, generated 2026-06-11): `co
|
|||
- `.kiro/steering/roadmap.md` — dependency ordering; superseded item "Quorum-ACK gate before config restart" residual
|
||||
- `.kiro/steering/ha-async-io.md`, `testing.md`, `error-handling.md` — threading, TDD, errnox conventions
|
||||
- Raft paper §5.3 (commit rule) — via umbrella; this spec adds no new consensus logic
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
> **Mode**: POST-IMPLEMENTATION GAP AUDIT. All tasks 1.1–7.2 are `[x]` (phase=implemented). This pass re-verifies every EARS acceptance criterion against the current on-disk controller source (codegraph + grep, `persistent_ha` working tree) rather than against the design intent, and records residual/deferred items the implementation explicitly left open.
|
||||
> **Verdict up front**: **CLOSED** — 19/19 acceptance criteria MET in code. The only open items are pre-declared, out-of-boundary residuals (bug-448 ref leak, bug-451 RUSAGE OOB, the CTRL_PRIMARY_COMMITTING fact-rejection deferred to ha-single-leader-role, and the WORKER_UPDATE-via-heartbeat WAIT-classification refinement). None block this spec's contract.
|
||||
|
||||
### Requirement-to-Asset Map (verified file:line)
|
||||
|
||||
| Req | Criterion (short) | Asset (verified) | Verdict |
|
||||
|-----|-------------------|------------------|---------|
|
||||
| 1.1 | success reply only after majority commit | `persist_commit_barrier` (persist.c:2069) called post-`job_ctrl` in `do_jobcontrol_op` (job.c:1023-1027), post-`queue_ctrl` in `do_queuecontrol_op` (queue.c:144), post-`worker_ctrl` in `do_hostcontrol_op` (worker.c:712-716), post-AFTER_EVENT in `handle_worker_join` (join.c:130) — reply (`xce_reply`) strictly after the barrier in every case | **MET** |
|
||||
| 1.2 | commit timeout → error reply | barrier returns `-ETIMEDOUT` (persist.c:2105 → `commit_submit(...,COMMIT_MODE_WAIT,timeout)`), `reply.status = rc` in all four handlers (job.c:1029, queue.c, worker.c:718); `timeout_ms==0` borrows `p->config.sync_timeout_ms` (persist.c:2104-2107) | **MET** |
|
||||
| 1.3 | not-ready rejection → error reply | engine `-XCE_ERR_NOT_READY` passes through barrier unmodified (persist.c:2105 returns engine code); handler maps to `reply.status` | **MET** |
|
||||
| 1.4 | no success-interpretable reply pre-commit | handlers send exactly one `xce_reply` after the barrier; integration test `test_commit_barrier_rollout.c` pins "no success reply while entries exceed committed_lsn" (task 3.1/7.2) | **MET** |
|
||||
| 2.1 | every WAL type classified WAIT/RETAIN | `wal_classify` (persist.c:2004-2054) total switch over all 13 `wal_entry_type` values incl. WAL_NOOP row (persist.c:2037, pre-declared); consulted in all 3 hook fns (`persist_job_hook_fun`/`persist_queue_hook_fun`/`persist_worker_hook_fun`, codegraph callers) | **MET** |
|
||||
| 2.2 | unclassified type fails fast / fact-fallback | default arm: `ERROR(...)` log + `assert(0)` under `XCE_TESTING` + `return WAL_MODE_RETAIN` production fallback (persist.c:2045-2053) | **MET** |
|
||||
| 2.3 | dispatch ACK is receipt-only | `handle_jobdispatch`/`dispatch_a_job_` produce no WAL (research §3); `do_set_preason`/`do_clr_preason` reply-first then RETAIN-append (job.c:1150/1178) — receipt verdict pinned by task 3.3 tests | **MET** |
|
||||
| 2.4 | exhaustive reply-site verification | task 7.1 audit (tasks.md Impl Note 7.1): ZERO request-shaped stragglers; 8 category-A barrier-gated, 6 category-B receipt-ACK, ~29 category-C sites enumerated and committed | **MET** |
|
||||
| 3.1 | facts appended, retried, no timeout | RETAIN types → `persist_wal_async` + `commit_submit(...,COMMIT_MODE_RETAIN,0)` (returns immediately, no timeout); hook callers get local-append rc only | **MET** |
|
||||
| 3.2 | facts accepted during quorum loss | `ha_dispatch_gate` CTRL_PRIMARY no-quorum allowlist adds `OPCode_notify/update/setpreason/clrpreason` (ha.c:2099-2102) before the `-XCE_ERR_NOT_READY` default (ha.c:2105) | **MET** |
|
||||
| 3.3 | uncommitted facts never discarded | log-is-the-queue: pending purely by `lsn > committed_lsn`; no drop path in the RETAIN seam | **MET** |
|
||||
| 3.4 | facts commit after quorum restore, no redelivery | `committed_lsn` advances over retained suffix via `commit_on_ack`; task 4.3 integration test asserts no re-delivery, no timeout applied | **MET** |
|
||||
| 4.1 | config confirmed only after majority | `do_upgrade_config_with_bytes` (config_repl.c:1115) calls `push_config_to_standbys` (config_repl.c:941) which returns barrier rc; success ACK (config_repl.c:1133-1142) only when `rc == 0` | **MET** |
|
||||
| 4.2 | config timeout → failed to requester | `rc != 0` → `send_upgrade_error(ep,req,c,rc,...)` then `return rc` (config_repl.c:1118-1126), forwarding `-ETIMEDOUT` | **MET** |
|
||||
| 4.3 | unacked config never committed | `committed_config_tag` advance is inside `push_config_to_standbys`'s success branch only (config_repl.c:963-969, atomic_store gated on majority) | **MET** |
|
||||
| 4.4 | failed upgrade does not restart | failure path returns before `restart_requested` store + `cbuf_put(&c->wbuf)` (config_repl.c:1159-1161); restart unreachable on failure | **MET** |
|
||||
| 5.1 | duplicate transition is logged no-op | `mark_job_finish` duplicate-on-terminal → `INFO(...)` no-op (job.c:702); `mark_job_update` no-op update → `INFO` skip (job.c:810) — no AFTER hook, no WAL | **MET** |
|
||||
| 5.2 | terminal→non-terminal rejected with WARN | shared `job_state_is_backward` predicate (job.c:39); `mark_job_start` rejects backward START (job.c:632-633 WARN); `mark_job_update` rejects backward (job.c:803-804 WARN) | **MET** |
|
||||
| 5.3 | replay-once equivalence | guards make RETAIN-retry / `restore_job_state` replay converge to single-application state; double-apply test (task 6.1) green | **MET** |
|
||||
|
||||
**Score: 19/19 MET.**
|
||||
|
||||
### Supporting-mechanism confirmations
|
||||
- **Legacy walAck retirement (design goal, Req 1.x)**: `repl_wait_majority_ack` (repl.c:422) is now configAck-only; walAck delegates unconditionally to `commit_submit` (repl.c:657-659 comment + `repl_set_commit_engine` repl.c:660). configAck branch hardened: `acked_config_lsn >= push_tag` (repl.c:500-501), majority `1 + covered > n_peers/2` over `n_peers` (repl.c:511), `n_peers <= 1` immediate (repl.c:470-472), zero-cover multi-member → timeout (repl.c:529). **MET / no residual.**
|
||||
- **Config-ACK production ingestion (remediation r1 / former CRITICAL F1)**: `repl_on_config_ack` now has live producers — `repl_ack_decode_and_dispatch` decodes `replconfigpushreplymsg` on the promoted-vote ep (ha.c:3403-3549) and a defensive `cluster.c:630-640` OPCode_reply ingestion. Closes the "bricked multi-member config upgrade" hole. **MET.**
|
||||
- **Barrier-off-iomgr (remediation r1 / former CRITICAL F2)**: `push_config_transport_only` (config_repl.c:879, non-blocking) serves all best-effort ha.c/join.c streaming sites; `push_config_to_standbys` (transport+barrier) has exactly one caller in vtask context (`do_upgrade_config_with_bytes`). **MET.**
|
||||
|
||||
### Residual / Deferred Gaps (all pre-declared out-of-boundary — none block CLOSED)
|
||||
|
||||
| Item | Tag | Where | Status & Rationale |
|
||||
|------|-----|-------|--------------------|
|
||||
| bug-448: `job_ctrl` reference leak | **Constraint** | `reference(j)` at job.c:348 on the not-started branch; normal exit never `dereference`s | NOT FIXED. Pre-existing refcount leak surfaced by task 3.2 adversarial review; explicitly out of this spec's boundary (no behavior in the commit contract). Future fix. |
|
||||
| bug-451: RUSAGE va_list OOB read | **Constraint** | `update_job_rusages` fires `HOOK_DETAIL_JOB_RUSAGE_UPDATE` with `xce_job_rusages_t*` (job.c:1046/1093-1094); `persist_job_hook_fun` default arm reads vararg 2 as `job_t*` → 4-byte OOB read at offset 56, garbage `state` in every WAL_JOB_RUSAGE payload | NOT FIXED. Reviewer-verified byte-precise; declared a boundary item at task 3.3. Production fix candidate = RUSAGE-specific va_list signature in the persist arm. Does not affect any acceptance criterion (jobid lines up; RUSAGE is a fact, retained either way). |
|
||||
| CTRL_PRIMARY_COMMITTING still rejects facts | **Missing (deferred)** | `ha_dispatch_gate` COMMITTING branch intentionally untouched (design Non-Goals; ha.c) | DEFERRED to ha-single-leader-role, which deletes the transitional state wholesale. During the brief COMMITTING window facts can still be rejected — known residual, hard-ordered downstream. |
|
||||
| WORKER_UPDATE WAIT-classification too broad | **Constraint** | `mark_worker_back` (worker.c:589) via heartbeat sweep `check_worker_timeout_` (cluster.c:1288) produces a WAIT-classified WAL_WORKER_CTRL/UPDATE with NO `xce_reply` | NOT a 7.1 violation (no reply ⇒ cannot confirm-before-commit; un-barriered WAIT behaves as RETAIN — commits via ACKs). Future-refinement candidate: narrow WORKER_UPDATE WAIT to the request-bearing detail. Carry to ha-single-leader-role / hardening pass. |
|
||||
| Half-pushed config on disk after failed upgrade | **Constraint** | `do_upgrade_config_with_bytes` saves to disk + bumps generation before the barrier; failure leaves the new config on the primary's disk but cluster-uncommitted | Accepted (design Decision "Failed config upgrade does not restart"). Reconciled by existing OP_replConfig anti-entropy (out of boundary). Operator sees explicit error; tag stays uncommitted. |
|
||||
|
||||
### Integration Risks (forward-looking, not gaps in this spec)
|
||||
- **ha-single-leader-role hard ordering**: must land AFTER this spec (it removes CTRL_PRIMARY_COMMITTING and adds WAL_NOOP). The classification guard (persist.c:2049) already forces WAL_NOOP to be classified — pre-declared row at persist.c:2037 (RETAIN) means the totality sweep will not trip when the type is wired. Low risk.
|
||||
- **worker-rejoin-resync**: shares the `mark_job_*` idempotency guards (job.c:39/632/803). Revalidation trigger: re-verify the backward predicate against resync replay ordering when that spec lands.
|
||||
- **New error codes to clients** (`-ETIMEDOUT`/`-XCE_ERR_NOT_READY` on control ops): deliberate CP behavior; flows through existing reply envelopes; caller retry semantics unchanged. Confirmed by design + reply.status wiring.
|
||||
|
||||
### Open-Work Options (only for the deferrable refinements — the spec itself needs none)
|
||||
|
||||
bug-451 (RUSAGE OOB) is the only residual with a correctness flavor inside this spec's files, so it carries options:
|
||||
- **Option A — Fix now (RUSAGE-specific va_list arm in `persist_job_hook_fun`)**: read vararg 2 as `xce_job_rusages_t*` for `HOOK_DETAIL_JOB_RUSAGE_UPDATE`. *Effort: S (≈1 file, 1 arm, 1 ASAN test). Risk: Low — isolated to the WAL_JOB_RUSAGE payload encode; no contract change.*
|
||||
- **Option B — Defer to a dedicated hardening pass / worker-rejoin-resync**: keep the boundary call as recorded. *Effort: 0 now. Risk: Low-Medium — OOB read persists in production (4-byte over-read, no crash observed; garbage state field in a fact payload that is retained regardless).*
|
||||
- **Option C — Fold into ha-single-leader-role sweep** (which already touches WORKER_UPDATE classification). *Effort: S, batched. Risk: Low, but couples an unrelated correctness fix to a larger spec's timeline.*
|
||||
|
||||
**Recommendation**: Option A for bug-451 (cheap, isolated ASAN-clean fix); leave bug-448 and the COMMITTING/WORKER_UPDATE items as recorded boundary residuals. The spec's own 19/19 contract is CLOSED — no spec-internal open work.
|
||||
|
|
|
|||
|
|
@ -16,19 +16,36 @@ Decoded the raw WAL bytes at `/tmp/xce-ha-deploy/controller/ctrl-{1,2,3}/data/wa
|
|||
|
||||
The committed prefix (LSN 2–12 = the lost jobs) is **physically absent** from both live nodes; it exists only on the down node. `snap/` dirs are **empty** on all nodes (no snapshot).
|
||||
|
||||
## DECISIVE EVIDENCE 2 — catch-up streaming DROPS payloads (bug-847, REPRODUCED 2026-06-20)
|
||||
A live restart of ctrl-1 (kill ctrl-1 → restart) showed it rejoin and catch up to the cluster length `[1..14]` with **correct LSN/type/CRC chain but EMPTY payloads** — `waldump` reports every JOB_SUBMIT as `PLEN=0 PCRC=00000000 jobid=?`, while ctrl-2/ctrl-3 still hold full 872-byte payloads. So the WAL is **structurally intact but content-empty** — silent corruption that passes every check that doesn't decode the payload (the `PCRC` column is what exposed it). This, not just the missing-prefix gap, is a concrete committed-job-loss mechanism: if the empty-payload node ever leads, the empty entries become authoritative.
|
||||
|
||||
**Root cause (reproduced in a unit test):** the catch-up batch wire path (`xce_wal_batch_encode_raw`/`wal_entry_to_wire` in `src/lib/request.c`) builds a `WalCatchupEntry` struct-list whose per-element `payload :Data` is allocated via `capn_new_list8(cs=root_seg,…)`. When the batch outgrows its first capnp segment, the per-element payload `Data` pointer is **lost across the segment boundary** → on decode `extract_data_bytes(we.payload)` reads `CAPN_NULL` → `raw_payload_len=0` → the receiver writes a payload-less entry. **`test_wal_batch_roundtrip_large_payloads` (14×872B) FAILS "Expected 872 Was 0"; the 2-entry round-trip PASSES** — the threshold is message SIZE (segment overflow), not entry count. **Live replication (`repl_broadcast`) carries payloads fine — only the catch-up/rejoin path corrupts**, which is exactly why loss surfaces only after a node restarts and catches up. Candidate fix: carry all sub-entry payloads as ONE top-level capnp `Data` blob + lengths (single `Data` round-trips correctly), sidestepping per-element cross-segment pointers. RED test is in `src/lib/testsuite/test_wal_wire.c`.
|
||||
|
||||
## ROOT CAUSE (mechanism, code-grounded)
|
||||
A node that (re)joins with an **empty WAL** is brought up to the cluster's **current committed watermark by WAL streaming `[watermark..committed]` — NOT from LSN 1 — and there is no snapshot to backfill `[1..watermark-1]`.** So it ends up with a permanent prefix gap. When the failover test eventually has the only full-log node down while the others rejoined at high watermarks, the committed jobs are gone for good.
|
||||
|
||||
Why the `seg-...001` file contains LSN 16: a node booting empty defaults `current_lsn=1` (`persist.c:429`); the first write lazily calls `open_new_segment` which names the file by `current_lsn` → `seg-...001` and **`O_TRUNC`s** it (`persist.c:145`); catch-up then writes entries carrying explicit high LSNs (16…) from offset 0. So "writes from the beginning of the file" = **booted empty + caught up from the watermark**, not an in-place overwrite of recovered data. (`replay_wal` at `persist.c:1422` is correct — it opens existing segments **read-only** and advances `current_lsn` past what it recovers.)
|
||||
|
||||
## OPEN FORKS (what to settle next)
|
||||
1. **Test procedure — clean vs preserve.** `scripts/cleanup.sh` does `rm -rf .../data/*`; `scripts/start.sh` only creates `data/` if missing (preserves). The on-disk files (named `001`, starting at 16/13) match the **cleaned** case. **Confirm with the user whether the failover test cleans data dirs before restart or restarts with data intact.**
|
||||
1. **Test procedure — clean vs preserve.** `scripts/cleanup.sh` does `rm -rf .../data/*`; `scripts/start.sh` only creates `data/` if missing (preserves). The on-disk files (named `001`, starting at 16/13) match the **cleaned** case. **Confirm with the user whether the failover test cleans data dirs before restart or restarts with data intact.** (Still worth confirming for the exact trigger ordering, but no longer the architectural question — see VERDICT.)
|
||||
- If preserved + node still returns gapped → **recovery bug** (`replay_wal` failing to restore).
|
||||
- If cleaned → **catch-up never backfills the prefix + no snapshot** = the architectural gap below.
|
||||
2. **Catch-up start point (the actual bug site, NOT yet read).** Trace the `CtrlHello` handshake: does a rejoining standby advertise its **real `applied_lsn`** (fresh ⇒ 0 ⇒ should stream from 1) or a **learned/committed watermark** (⇒ streams from 13/16)? `CtrlHelloRequest.appliedLsn @2` is the value the primary streams `[applied+1..committed]` from. This single field decides the fix: "stream from 1 / install a snapshot first" vs "advertise real applied, not learned committed."
|
||||
3. **Snapshots.** `snap/` is empty everywhere. Is snapshot *creation* implemented/triggered in this build, or only `load_snapshot` (`persist.c:1266`, format `"XCESNAP\0"` + 8-byte lsn header — looks like a STUB carrying only an LSN, NO entities)? If snapshots never carry job entities (the "entity-bearing snapshots" backlog item in ha-convergence-recovery), then ANY WAL pruning/empty-rejoin below committed is unrecoverable BY DESIGN. **Strong candidate for the architectural root.**
|
||||
- If cleaned → **catch-up never backfills the prefix + no snapshot** = the architectural gap (CONFIRMED below).
|
||||
2. **Catch-up start point — ✅ SETTLED (2026-06-20).** Traced. `handle_ctrlhello` (`config_repl.c:1436`) is connection-setup ONLY; catch-up is driven by the heartbeat decision loop. The standby reports its **REAL `applied_lsn`** — `ha_process_standby_state(ha, peer_idx, sr.config_tag, sr.applied_lsn, …)` at `ha.c:3518`, where `sr.applied_lsn` is the standby's own state-report value (an empty standby reports 0). The decision `ha_evaluate_repl_action` (`ha.c:1811`) is CORRECT: `standby_lsn+1 < oldest_wal_lsn → SEND_SNAPSHOT`, `else standby_lsn < committed → STREAM_WAL`. So advertise is NOT the bug — a fresh standby correctly asks for a snapshot when the primary's WAL no longer starts at 1.
|
||||
3. **Snapshots — ✅ SETTLED: THIS IS THE ROOT CAUSE (2026-06-20).** `write_snapshot` (`persist.c:1051`) writes a **16-byte SENTINEL**: `"XCESNAP\0"` (8) + `checkpoint_lsn` (8) — its own comment says *"Full Cap'n Proto serialization is Task 11."* It carries **ZERO job/queue entities.** `load_snapshot` (`persist.c:1267`) just does `p->checkpoint_lsn = p->current_lsn = stored_lsn` — it **advances the watermark past entries it never received, restoring nothing.** And `persist_checkpoint` → `trim_wal_segments` (`persist.c:1090`) **deletes WAL segments below the checkpoint.** So checkpoint = "trim the committed prefix + write an empty backup": a WAL-eating loop with no recoverable safety net. This is the "entity-bearing snapshots" gap (ha-convergence-recovery backlog) — and it makes any empty-rejoin/prefix-trim below committed **unrecoverable BY DESIGN.**
|
||||
|
||||
## VERDICT (2026-06-20) — architectural root confirmed, code-grounded end-to-end
|
||||
The loss is **not** an advertise/start-point bug and **not** a recovery bug. It is the **absence of entity-bearing snapshots** combined with **prefix-trimming checkpoints**. Full mechanism:
|
||||
1. A fresh/empty member (applied_lsn=0), or any member catching up against a primary whose own `oldest_wal_lsn > 1` (its prefix was trimmed by a prior checkpoint).
|
||||
2. `ha_evaluate_repl_action` → `SEND_SNAPSHOT` (when `1 < oldest_wal_lsn`) or `STREAM_WAL` from the high watermark.
|
||||
3. SEND_SNAPSHOT path: `persist_checkpoint` writes the **sentinel** (no entities) + **trims the prefix WAL**, then ships the sentinel; receiver `persist_load_snapshot` sets `current_lsn = checkpoint_lsn` with **no entity restore**.
|
||||
4. STREAM_WAL path: `ha_stream_wal_to_standby` streams `[standby_lsn+1 .. committed]`; `ha_apply_wal_catchup_entry`/`ha_apply_single_entry` **physically persist** those entries (consistent with ctrl-3's on-disk seg holding LSN 13–18) — but the prefix `[1..oldest-1]` is **never streamed and has no snapshot to backfill it.**
|
||||
5. Net: the member's watermark reaches `committed` while the committed-prefix job entities are gone. **Permanent loss, BY DESIGN.**
|
||||
|
||||
**Fix layer (recommendation — option (b), question-the-architecture):** implement **entity-bearing snapshots** — `write_snapshot` Cap'n-Proto-serializes all live job/queue entities at `checkpoint_lsn`; `load_snapshot` deserializes + `restore_job`/restore_queue them BEFORE advancing `current_lsn`; and `trim_wal_segments` must NOT trim below the LSN covered by a durable entity-bearing snapshot (install-snapshot-before-stream on the receive side). Option (a) "stream `[1..committed]`" only works while some node still retains LSN 1 (false after any trim), so it is a partial mitigation, not the fix. This is the **ha-convergence-recovery** "entity-bearing snapshots" item; route the fix there (or a new `ha-entity-snapshots` spec) under the ha-commit-durability umbrella.
|
||||
|
||||
## SECONDARY FINDINGS
|
||||
- **Double `WAL_JOB_SUBMIT` per job (bug-845, 2026-06-20, via waldump).** Every committed job appears as TWO consecutive entries with the SAME jobid (ctrl-1: 2,3 / 4,5 / … ; ctrl-3: 14,15 ; ctrl-2: 17,18). **Ruled out array expansion:** `job_submit` assigns array members DISTINCT ids (`base_id+i`, job.c) and the `Job` capnp has a single jobid field, so identical-jobid pairs are NOT two array tasks — a single submit is double-appending. **CONFIRMED a literal byte-for-byte double-append** by the enhanced waldump (payload-CRC + subtime columns, 2026-06-20): each pair is byte-identical — same PCRC (LSN 2,3 both `6d79074e`; 4,5 `c25ac163`; …) and same `subtime` (e.g. 1781959730), so all 12 ctrl-1 submits = 6 jobs × 2. A failover retry would re-stamp `subtime=time(NULL)` and differ → **retry definitively ruled out; it's the SAME encoded `job_t` written twice in one submit flow.** It is replicated *consistently* (F=…555020288 at 14,15 on BOTH ctrl-1 and ctrl-3; G at 17,18 on BOTH ctrl-2/3), so not itself divergence, but it doubles LSN consumption and is a latent cross-node divergence risk if the duplicate count ever varies by primary. Next: trace `handle_jobsubmit`→`persist_job_hook_fun` AFTER_EVENT loop (jlist holding the job twice, or a double hook-fire).
|
||||
- **Raft cross-node consistency check (user request, 2026-06-20).** Per-LSN comparison of the three dumps: at every OVERLAPPING index the entries AGREE — ctrl-1∩ctrl-3 = {13:NOOP, 14:F, 15:F}; ctrl-2∩ctrl-3 = {16:NOOP, 17:G, 18:G}. So **State Machine Safety (no two members hold different entries at the same index) is NOT violated** in this snapshot; no job sits at different LSNs on different nodes. The broken guarantee is **Leader Completeness / durability**: jobid E=…394588160 was committed+acked at LSN 11–12 (present on ctrl-1) but is ABSENT from the surviving majority (ctrl-2/3) — a committed entry lost (exactly the root-cause mechanism: it was below the survivors' empty-rejoin catch-up start, with no entity snapshot to backfill it). Conversely G=…630517760 is on the survivors but absent on ctrl-1, which was already down before G — losing an entry a crashed minority-node never had is Raft-legal. So the cluster ended with two committed histories sharing only the F segment: ctrl-1's {…,E@11-12,F@14-15} vs survivors' {F@14-15,…,G@17-18}.
|
||||
- **`term=0` on every WAL entry.** `p->term` (the `hdr.term` source) is **never assigned** the election term anywhere (grep for `p->term =` / `->term =` in persist.c/ha.c returns nothing). So per-entry term is dead. XCE matches logs via the `prev_crc` chain, so possibly benign — but it means **no term-based divergence detection**. Worth a hard look (could let divergent same-LSN entries be treated as matching).
|
||||
- **Cluster cannot hold a primary** under the test: peer connect `-111` (ECONNREFUSED), RDHUP, rapid term churn 2→5 in ~7 min, elections logged `safe_committed_lsn=X (from 1 granted voters)`. Logged separately in `ha-convergence-recovery/research.md` (2026-06-20 live finding).
|
||||
- **Crash fixed en-route (unrelated):** `xce_strerror_r` global-buffer-overflow on HA error codes — bug-827, fixed in `src/lib/utils.c` (commit 1aa0761).
|
||||
|
|
@ -45,14 +62,39 @@ The loss **predates** it, so it is **not the origin**. But two real concerns it
|
|||
- Deployed binary includes controller-status (responderTime) + ha-heartbeat-committed (the `ha.c:4967 "learned committed_lsn=X from heartbeat"` logs are the consume path).
|
||||
|
||||
## NEXT STEPS
|
||||
1. **Build the WAL/snapshot inspection tooling in `xce_test`** (in progress — see plan below) so clean-vs-preserve and gap-location become a 30-second check each run.
|
||||
1. **Build the WAL/snapshot inspection tooling in `xce_test`** — ✅ `waldump` DONE (2026-06-20). `snapdump`/`walcompare` still TODO. Gap-location is now a one-liner per run.
|
||||
2. **Read the CtrlHello catch-up advertise/start-point** (`config_repl.c` handle_ctrlhello + the standby's CtrlHelloRequest build) — Open Fork #2.
|
||||
3. **Confirm whether snapshot creation exists** or is a stub — Open Fork #3 (likely the architectural root: no entity-bearing snapshot ⇒ no prefix backfill).
|
||||
4. Decide fix layer: (a) full-WAL stream `[1..committed]` when a node has no snapshot, (b) implement entity-bearing snapshots + install-before-stream, or (c) prevent empty-rejoin from advancing committed without holding the entries. Likely a **question-the-architecture** moment (catch-up assumes a snapshot safety net that does not exist).
|
||||
|
||||
## DECISIVE EVIDENCE 3 — bug-851 REPRODUCED & ROOT-CAUSED on an owned cluster (2026-06-21)
|
||||
|
||||
Stood up a fresh local 3-node cluster (`/xce-deploy-ha`, binary built with the 848/849/850 fixes) and drove two scenarios with full log access. Artifacts: `repro-851/{ctrl-1,ctrl-2,ctrl-3}.log` + `waldump-ctrl-{1,2,3}.txt`.
|
||||
|
||||
**Scenario 1 — clean single failover: PASS (848/849/850 validated end-to-end).**
|
||||
6 jobs → identical `1..13` on all 3 (matching PCRCs). Kill primary ctrl-1 → ctrl-3 promoted: `cluster_init_phase2: persist_recover done — recovered durable_floor=13 … applied_lsn=14 advertised` (bug-849 recovery-on-promotion + bug-848 advertise-real-LSN both fire). 4 jobs on new primary → ctrl-2 converges `14..22`. Restart old primary → recovers `1..13`, catches up `14..22`. **Final: all three WALs byte-identical `1..22`, identical PCRC fingerprint `cc5d361e…`, 0 gaps.**
|
||||
|
||||
**Scenario 2 — chaotic leadership churn (3× kill-the-primary): FAIL — silent committed-prefix divergence.**
|
||||
After 3 rounds, all three report `committedLsn=54`, all `lsn range 1..54`, 0 gaps, CRC intact — **but the logs DIVERGE at LSN 45–54**:
|
||||
- ctrl-1: `45=NOOP`, `46..53 = faead246/be0059bf/fe7a3d0b/37d54e4d` → fp `bfeb26ad…`, 25 jobids
|
||||
- ctrl-2: `45=NOOP`, `46..53 = d32ac747/97c04cbe/d7ba280a/1e155b4c` (DIFFERENT jobs) → fp `216af434…`, 25 jobids
|
||||
- ctrl-3: `45 = 4c6d24e6` (a JOB, no noop), `45..52 = 4c6d24e6/0887af1f/48fdcbab/8152b8ed` → fp `865304ec…`, 26 jobids
|
||||
|
||||
Three different "committed" tails, each node reporting them committed. This violates the Raft committed-prefix invariant (a committed entry must be byte-identical on every member).
|
||||
|
||||
**ROOT CAUSE (code-grounded, three compounding defects):**
|
||||
1. **Term not monotonic across the cluster.** Election term = *locally-persisted* term + 1; PreVote explicitly does NOT persist/propagate ("starting PreVote round … no persist"). Under chaotic kills a leader wins term N, appends its own-term noop + jobs, and is killed **before term N is durably propagated to a majority**. A restarted node recovers a *stale* term and elects at the **same** N+1. Logs show **two leaders at term 3** (ctrl-3 @08:40, ctrl-1 @08:47:39) and **two leaders at term 4** (ctrl-2 @08:47:52, ctrl-3 @08:48:04) — each appending its own-term noop at the **same LSN 45**. Persisted terms ended split: ctrl-1=4, ctrl-2=4, ctrl-3=3.
|
||||
2. **The WAL entry term field is DEAD (always 0).** `wal_entry_hdr_t.term` exists (wal_types.h:30) and the replication envelope carries `term` (typemap.h:381), but the writer does `hdr.term = p->term` (persist.c:588/726/1040) and **`p->term` is NEVER assigned from `ha->term` anywhere in the controller** (zero matches). Every entry, every term, persists `term=0`. waldump confirms term=0 cluster-wide.
|
||||
3. **No (term,lsn) reconcile → divergent tails never truncated.** Because the WAL has no usable per-entry term, rejoin-reconcile (`ha_convergence.c`) cannot do Raft log-matching: entry 45@ctrl-1 (term 3) and 45@ctrl-2 (term 4) look identical (both term 0, both JOB_SUBMIT/NOOP), so the divergent suffix is taken as already-present and is never overwritten. `ha_compute_safe_committed_lsn` then advances committed (44, 53) "from 1 granted voter" over content that other members never received.
|
||||
|
||||
Net: under leadership churn, acked entries written by a short-lived leader become a permanent divergent tail, and `committed` advances over it → silent loss + cross-node divergence. This is distinct from the catch-up payload-drop (bug-847) and the empty-rejoin gap (bug-828); it is a **Raft-safety gap in term establishment + WAL term recording**.
|
||||
|
||||
**Keystone fix candidate (smallest correct lever):** stamp `hdr.term` from the live raft term (`ha->term`) on the primary append path + carry it through the replication envelope + persist it, THEN make reconcile truncate any local suffix whose (term,lsn) does not match the leader's — standard Raft log-matching. Must be paired with making the election term durably-established-before-acting (no two leaders per term). **HA-core + persistence → full spec workflow required (per CLAUDE.md); not a quick patch.**
|
||||
|
||||
**Secondary (separate) finding — client primary-discovery is positional, not failover-correct.** With `masters = [m0, m1, …]`, the client (`xcecli_ctrlstatus_primary`, client.c:403) only reaches the primary if `m0` IS the primary: a standby answers `primaryName=""` ("unknown" by design, ctrlstatus.c:104) so no redirect hint is captured, and the inner `xcecli_ctrlstatus` SUCCEEDS on the reachable non-primary so transport failover never rotates. Confirmed: `9993`(primary-first)→ok 45ms; `9992,9993`(standby-first)→"no primary"; `9991,9993`(dead-first)→"no primary"; `9993,9991`→ok. Also the dead node returns `ctrlstatus failed: Invalid argument` (bug-846 EINVAL). This is what made earlier remote sessions read as "committed frozen" — the client was pinned to a stale/standby node. Fix options: (a) standby tracks+reports the primary name it heartbeats with (enable redirect), or (b) client rotates to the next master on a reachable-but-non-authoritative reply.
|
||||
|
||||
## TOOLING PLAN (xce_test enhancement — requested by user)
|
||||
Add read-only debug commands to `xce_test` (`src/controller/testsuite/test.c`; binary links `persist.c`/`job.c` per `controller/meson.build`, so the capnp `restore_job`/`read_Job` decode and `wal_types.h` are available):
|
||||
- `waldump <data_dir|wal_dir|seg_file>` — per node/segment, list every entry `lsn | term | type | payload_len | jobid(submits, via capnp `read_Job`/`decode_Job`)`, flag **LSN gaps** (non-contiguous / missing prefix) and the recovered tip.
|
||||
- `snapdump <snap_dir|snap_file>` — decode the `XCESNAP` header (lsn) and report whether it carries entities (it likely does NOT — confirms Fork #3).
|
||||
- `walcompare <dirA> <dirB>` — diff two nodes' logs by `lsn → crc/jobid` to flag divergence/gaps instantly.
|
||||
Start with `waldump` (highest value; decoder logic already proven in Python). Register in the `client_commands`/`top_commands` table (`test.c:175/207`).
|
||||
Read-only debug commands in `xce_test`. **NOTE (correction):** `xce_test` does NOT link `persist.c`/`job.c` — it builds from `test.c` only + `libxce`. But `read_Job` is generated into `controller.capnp.c` which IS compiled into `libxce`, so the capnp decode is reachable; `waldump` adds `c_capnproto_dep`+`zlib_dep` to `xce_test_deps` and ships its own zero-alloc jobid decode (reads `struct Job.jobid` after `read_Job`, no `decode_Job`).
|
||||
- `waldump <data_dir|wal_dir|seg_file>` — ✅ **DONE 2026-06-20.** Per node/segment, lists every entry `lsn | term | type | payload_len | jobid`, flags **LSN gaps** (non-contiguous / missing prefix) + the CRC recovered tip; nonzero exit on gap/CRC-break so it doubles as an E2E assertion. Impl: `src/controller/testsuite/waldump.{c,h}` (module), `cmd_waldump` wired into `top_commands` in `test.c`; unit test `test_waldump.c` (5 cases, hand-built raw segments). **Validated live** against `/tmp/xce-ha-deploy/controller/ctrl-{1,2,3}` — reproduced the exact bug table (ctrl-1 full 1–15; ctrl-2 missing prefix 1–15; ctrl-3 missing prefix 1–12). Also surfaced a NEW anomaly: each JOB_SUBMIT appears as a **consecutive duplicate pair** (same jobid at LSN N and N+1) — worth a follow-up.
|
||||
- `snapdump <snap_dir|snap_file>` — TODO. Decode the `XCESNAP` header (lsn) and report whether it carries entities (it likely does NOT — confirms Fork #3).
|
||||
- `walcompare <dirA> <dirB>` — TODO. Diff two nodes' logs by `lsn → crc/jobid` to flag divergence/gaps instantly.
|
||||
|
|
|
|||
73
.kiro/specs/ha-commit-durability/brief.md
Normal file
73
.kiro/specs/ha-commit-durability/brief.md
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
# Brief: ha-commit-durability
|
||||
|
||||
## Problem
|
||||
A 3-node HA controller failover test **permanently lost committed jobs**. The client submitted 10 jobs across several forced primary kills; the 7 earliest (JobIds `0-0-1781708993..1781709035`, WAL LSN ≤20) — which had been replicated and majority-ACKed, and survived the *first* failover — were gone by the end, while only the 3 latest survived. Losing an acknowledged, committed job is a correctness (data-loss) failure, the most severe class for this system.
|
||||
|
||||
## Current State
|
||||
The HA consistency machinery is implemented (commit engine, WAL replication, convergence/truncate-on-demotion, single-leader own-term gate, PreVote, vote-freshness bug-339). But **"committed" does not imply "durable on a majority"**:
|
||||
|
||||
- **WAL fsync is asynchronous** — a timer thread (`persist.c:173 fsync_worker`) fdatasyncs every `wal_sync_ms`. `persist_append` does **not** fsync before returning; the standby does **not** fsync before sending its `WalAck`.
|
||||
- **The commit engine advances `committed_lsn` on majority *ACK*** (`commit.c:303 commit_on_ack`, `repl_wait_majority_ack`). An ACK only proves the entry reached the OS page cache — it can be durable on **zero** disks at the moment it is declared committed.
|
||||
- **Recovery silently truncates to nothing.** When killed nodes restart, `persist_recover` hits a CRC chain break at a zeroed/never-fsync'd region (`ctrl-3.log:1597` `prev_crc=00000000` at offset 40) and stops replay at ~0 entries — but the node still becomes eligible.
|
||||
- **A hollow node wins and overwrites the good timeline.** With durability lost on the killed nodes and failover quorum collapsed to `required=1`, the freshness gate cannot fence the stale candidate. ctrl-3 became PRIMARY term=4 advertising `committed_lsn=21` over an empty job table (`ctrl-3.log:1612`); the node that *did* hold the data (ctrl-1, applied_lsn=22) converged **down** to it (`ctrl-3.log:1618`). LSN ≤20 discarded forever.
|
||||
|
||||
This violates the Leader-Completeness assumption documented in `src/controller/CLAUDE.md` ("committed entries are on a majority" presumes majority-ACK ⇒ durable-on-a-majority) and the TLA+ mapping `log[i] = WAL on disk`.
|
||||
|
||||
## Desired Outcome
|
||||
1. An operation reported committed is **durable on a majority of nodes** before it is declared committed — surviving simultaneous crash + restart of any minority.
|
||||
2. A node that cannot recover its WAL to a valid committed tip is **fenced from leadership** (fail-loud), never silently truncating to zero and then leading; the cluster never converges onto a timeline below its known committed high-water mark.
|
||||
|
||||
## Approach
|
||||
Two coupled requirements (chosen scope: *Durability + promotion fence*):
|
||||
|
||||
- **R1 Durable commit (fsync-before-ACK):** the standby `fdatasync`s the appended WAL bytes **before** emitting `WalAck`; the primary counts its own vote toward majority only after its local `fdatasync`. `committed_lsn` advances only on a **durable** majority. Preserve throughput via group-commit / batched fdatasync (amortize the sync across pending appends) rather than one sync per entry. Keep `wal_sync_ms` async path only for non-committing/best-effort writes, if any remain.
|
||||
- **R2 Recovery & promotion fence:** a WAL CRC/replay failure that prevents reconstructing a contiguous committed tip makes the node **ineligible to become primary** until it has caught up from a healthy peer (FATAL or fence, never silent truncate-to-zero-then-lead). A node must not win election or self-promote advertising a `committed_lsn` below the cluster's known committed high-water mark, even under degraded single-voter quorum.
|
||||
|
||||
Honor the user's fail-loud preference: surface the real durability/recovery failure rather than masking it.
|
||||
|
||||
## Scope
|
||||
- **In**: fsync-before-ACK on the standby apply path and primary self-count; durable-majority gating of `committed_lsn`; group-commit batching to bound the latency cost; fencing an un-recovered/corrupt-WAL node from promotion; preventing a sub-high-water-mark leader from being elected under degraded quorum; unit tests for each; the 3-node live E2E that reproduces this incident (kill-primary-under-load, assert zero committed-job loss).
|
||||
- **Out**: WAL wire format / ACK message shape (`ha-wal-replication`); per-LSN majority math (`ha-commit-engine` — consumed, the *durability precondition* of its ACK input is what changes); demotion truncate/self-restart and rejoin reconciliation of an *intact* log (`ha-convergence-recovery`); promotion no-op / own-term gate (`ha-single-leader-role`); raw vote-grant arithmetic (`raft-library` — this spec adds the durability+high-water *precondition*, not new vote math); entity-bearing snapshots (roadmap); client-side failover (`libxce-failover`).
|
||||
|
||||
## Boundary Candidates
|
||||
- Durable-commit seam: standby `fdatasync`-before-`WalAck` + primary self-fdatasync before majority self-count (the "ACK ⇒ durable" guarantee).
|
||||
- Promotion-eligibility seam: recovery-integrity + committed-high-water check gating `raft_cb_become_leader` / PreVote eligibility.
|
||||
|
||||
## Out of Boundary
|
||||
- Tuning the async `fsync_worker` cadence as a "fix" (it is the defect for committing writes, not a knob to turn).
|
||||
- Changing how a majority is *counted* (commit-engine) or how votes are *granted* (raft-library) beyond adding the durability/high-water precondition.
|
||||
- Worker-reported fact resync across restart (`worker-rejoin-resync`, roadmap backstop).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: `ha-commit-engine` (committed_lsn semantics it now gates on durable ACKs), `ha-wal-replication` (ACK/catch-up machinery this makes durability-correct), `persist.c` fsync infrastructure, `ha-prevote` + vote-freshness (bug-339) eligibility path it strengthens.
|
||||
- **Downstream**: `ha-convergence-recovery` E2E (its single-outcome guarantee depends on this), `component-failover-adoption` and any real failover deployment.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends/strengthens**: `ha-wal-replication` (durability precondition on ACK), `ha-commit-engine` (durable-majority), `ha-convergence-recovery` (assumes intact persisted committed state — this spec makes that assumption true).
|
||||
- **Adjacent (avoid overlapping)**: `ha-single-leader-role`, `raft-library`, `ha-prevote`.
|
||||
|
||||
## Constraints
|
||||
- C / gnu17, `src/controller/persist.c`, `commit.c`, `repl.c`, `ha.c`, `cluster.c`; TDD mandatory (tests in `testsuite/` before C changes).
|
||||
- Async-I/O rule: never block the iomgr thread — fdatasync must not stall heartbeat/vote handling; use the dedicated persist/fsync thread + group-commit and signal the commit waiter on completion.
|
||||
- Must verify against `raft/raft.tla` and the `src/controller/CLAUDE.md` Protocol Verification Checklist (commitIndex monotonic; leader holds all committed entries).
|
||||
- Live E2E needs `/xce-deploy-ha` (cluster at `/tmp/xce-ha-deploy`); reproduction harness = `xce_test` driving jobsub + primary kills. An automated soak reproduction now exists at `src/tools/test-commit-durability.sh` (rolling SIGKILL+restart under load, asserts every acked job survives).
|
||||
- Performance: durable-majority adds sync latency to the submit path — group-commit is required, not optional.
|
||||
|
||||
## Reproduction Findings (2026-06-17, from building the E2E)
|
||||
- **bug-661 is a RACE, not a deterministic failure.** `wal_sync_ms` is hardcoded to `DEFAULT_WAL_SYNC_MS = 10` ms (`config.h:86`) with **no CLI/env/config override**. The committed-but-not-fsync'd window is ~10ms, so a single-node SIGKILL+restart only loses a committed job when the kill lands inside that window AND the truncated node re-leads before catch-up repairs its WAL. The soak reproduced real loss once (4 committed jobs) but passed an 8-round run clean — confirming the rarity.
|
||||
- **Required test seam (NEW spec item): a configurable fsync interval** (e.g. `--wal-sync-ms`, plumbed into `persist_config_t.wal_sync_ms`). Setting it to several seconds makes every committed entry provably un-fsync'd at kill time, turning the racy soak into a **deterministic** RED. This seam is also the natural way to unit-test fsync-before-ACK. None exists today — add it as part of Requirement 1's testability (cf. the convergence spec's XCE_TESTING restart seam).
|
||||
- **Acceptance asymmetry**: a soak FAIL proves the bug; a soak PASS does not prove durability (the race may simply not have fired). After the fix, the soak must pass *deterministically regardless of timing* — that, plus the deterministic `--wal-sync-ms` test, is the real acceptance bar.
|
||||
- **Amplifier confirmed**: loss needs a killed+restarted node to re-lead with a truncated WAL *before* replication catch-up repairs it. The fix must therefore also fence a node that has not caught up to the committed high-water mark from leading (Requirement 2).
|
||||
|
||||
## Design Decision — durability mechanism (2026-06-17, with user)
|
||||
|
||||
**Invariant (user-stated, authoritative):** when the controller replies **ok** to a client, the operation's WAL entry MUST already be persisted (fdatasync'd) on a fault-tolerant majority — self + ⌊N/2⌋ standbys.
|
||||
|
||||
**Exact defect located:** `commit.c:majority_candidate_locked` counts the primary's own contribution as `cap = highest_appended_lsn(p)` — every *appended* (page-cache) entry, not every *fdatasync'd* one. On 3 nodes a commit needs self + 1 standby; the standby ACK is durable (`persist_wal_write_replicated:515` fdatasyncs before replying) but the primary's copy is **not** (it relies on the `fsync_worker` timer). So a "committed" entry is durable on **exactly 1 node** → zero durability fault-tolerance. That is bug-661.
|
||||
|
||||
**Chosen direction (user): synchronous persistence — eliminate `fsync_worker` as the durability mechanism.**
|
||||
- The submit path **already blocks off the iomgr thread** waiting for majority replication (`job.c:568-569`; `persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT)`), so adding the primary's own `fdatasync` on that path is liveness-safe — it does not stall heartbeats.
|
||||
- Minimal correctness change: (1) primary `fdatasync`s its entry before its self-contribution covers that LSN (symmetric with the standby path at `persist.c:515`); (2) `commit.c` self-contribution uses a `durable_lsn` (highest fdatasync'd), not `highest_appended_lsn`; (3) reply stays gated on the commit barrier → reply-ok ⇒ durable-majority.
|
||||
- **Throughput**: per-entry fdatasync. The dominant amplifier (a job array fanning out to N entries → N fsyncs) is removed by the roadmap item **job-array-single-wal-record** (one WAL record per array → one fsync). Cross-reference that item; this spec assumes it lands or is acceptable interim.
|
||||
- **Group-commit IS in scope (revised 2026-06-18 at design-review).** Initially deferred, but `/kiro-validate-design` found the highest-volume WAL path is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible, not covered by job-array-single-record — so a pure per-entry synchronous design penalizes the dominant writes. Decision (with user): repurpose `fsync_worker` into a demand-driven group-commit flusher (batch `fdatasync`, advance `durable_lsn`, wake commit-eval) as part of this spec. The roadmap "WAL group-commit batching" item is absorbed here.
|
||||
- **Open for design:** whether to delete the `fsync_worker` thread entirely or retain a minimal best-effort flush for any non-client-acked / local-only writes (e.g. checkpoint markers already fsync inline; audit `wal_classify` for any write that gates no reply). Confirm no client-acked op bypasses the commit barrier.
|
||||
164
.kiro/specs/ha-commit-durability/repro-851/ctrl-1.log
Normal file
164
.kiro/specs/ha-commit-durability/repro-851/ctrl-1.log
Normal file
|
|
@ -0,0 +1,164 @@
|
|||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ main] My name is ctrl-1
|
||||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
|
||||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
|
||||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
|
||||
[2026-06-21 08:37:35,671] [1217413] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=0)
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:35,674] [1217413] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:37:40,873] [1217413] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=0 current_lsn=1; applied_lsn=0 advertised for catch-up
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [raft_cb_become_follower] HA FSM (raft): → CTRL_STANDBY (term=1, first_boot=1)
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:50,478] [1217413] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 8ms (priority=0) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:37:50,679] [1217413] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=2, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:37:50,728] [1217413] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 2)
|
||||
[2026-06-21 08:37:50,728] [1217413] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=0 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=2, first_boot=0)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:37:50,738] [1217413] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:37:50,738] [1217413] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=1, writable_lsn gate armed
|
||||
[2026-06-21 08:37:53,924] [1217413] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1)
|
||||
[2026-06-21 08:37:53,925] [1217413] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1)
|
||||
[2026-06-21 08:39:27,321] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=2 < target=5 → streaming [2+1..5]
|
||||
[2026-06-21 08:39:27,321] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=3 target_lsn=5
|
||||
[2026-06-21 08:39:27,322] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [2+1..5] to standby
|
||||
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=3 < target=7 → streaming [3+1..7]
|
||||
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=4 target_lsn=7
|
||||
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [3+1..7] to standby
|
||||
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=7 < target=11 → streaming [7+1..11]
|
||||
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=11
|
||||
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..11] to standby
|
||||
[2026-06-21 08:39:27,922] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=4 < target=13 → streaming [4+1..13]
|
||||
[2026-06-21 08:39:27,922] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=5 target_lsn=13
|
||||
[2026-06-21 08:39:27,923] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [4+1..13] to standby
|
||||
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=8 < target=13 → streaming [8+1..13]
|
||||
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13
|
||||
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby
|
||||
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=5 < target=13 → streaming [5+1..13]
|
||||
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=6 target_lsn=13
|
||||
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [5+1..13] to standby
|
||||
[2026-06-21 08:39:28,523] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=9 < target=13 → streaming [9+1..13]
|
||||
[2026-06-21 08:39:28,523] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13
|
||||
[2026-06-21 08:39:28,524] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby
|
||||
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=6 < target=13 → streaming [6+1..13]
|
||||
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=7 target_lsn=13
|
||||
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [6+1..13] to standby
|
||||
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=10 < target=13 → streaming [10+1..13]
|
||||
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13
|
||||
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby
|
||||
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=7 < target=13 → streaming [7+1..13]
|
||||
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=13
|
||||
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..13] to standby
|
||||
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=11 < target=13 → streaming [11+1..13]
|
||||
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13
|
||||
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby
|
||||
[2026-06-21 08:39:29,525] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=8 < target=13 → streaming [8+1..13]
|
||||
[2026-06-21 08:39:29,525] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13
|
||||
[2026-06-21 08:39:29,526] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby
|
||||
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=12 < target=13 → streaming [12+1..13]
|
||||
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13
|
||||
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby
|
||||
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=9 < target=13 → streaming [9+1..13]
|
||||
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13
|
||||
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby
|
||||
[2026-06-21 08:39:30,326] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=10 < target=13 → streaming [10+1..13]
|
||||
[2026-06-21 08:39:30,326] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13
|
||||
[2026-06-21 08:39:30,327] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby
|
||||
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=11 < target=13 → streaming [11+1..13]
|
||||
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13
|
||||
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby
|
||||
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=12 < target=13 → streaming [12+1..13]
|
||||
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13
|
||||
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ main] My name is ctrl-1
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ cluster_init_phase1] HA: restored term=2 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
|
||||
[2026-06-21 08:43:44,327] [1224672] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=2)
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=2, raft_term=2, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=13 advertised for catch-up
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:43:45,250] [1224672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:43:45,250] [1224672] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=6)
|
||||
[2026-06-21 08:47:39,017] [1224672] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 122ms (priority=0) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:39,217] [1224672] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:47:39,262] [1224672] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3)
|
||||
[2026-06-21 08:47:39,265] [1224672] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=0)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:47:39,272] [1224672] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:47:39,272] [1224672] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed
|
||||
[2026-06-21 08:47:41,419] [1224672] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=48 → streaming [46+1..48]
|
||||
[2026-06-21 08:47:41,419] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=48
|
||||
[2026-06-21 08:47:41,420] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..48] to standby
|
||||
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=48 < target=53 → streaming [48+1..53]
|
||||
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=53
|
||||
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..53] to standby
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ main] My name is ctrl-1
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
|
||||
[2026-06-21 08:47:47,839] [1228504] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=3)
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up
|
||||
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=8)
|
||||
[2026-06-21 08:48:04,047] [1228504] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 159ms (priority=0) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:48:04,248] [1228504] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:48:04,295] [1228504] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4)
|
||||
[2026-06-21 08:48:04,295] [1228504] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=53 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:48:04,305] [1228504] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:48:04,307] [1228504] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=54, writable_lsn gate armed
|
||||
[2026-06-21 08:48:07,493] [1228504] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54)
|
||||
[2026-06-21 08:48:07,493] [1228504] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54)
|
||||
99
.kiro/specs/ha-commit-durability/repro-851/ctrl-2.log
Normal file
99
.kiro/specs/ha-commit-durability/repro-851/ctrl-2.log
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
[2026-06-21 08:37:36,163] [1217460] [ INFO] [ main] My name is ctrl-2
|
||||
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
||||
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
||||
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
||||
[2026-06-21 08:37:36,173] [1217460] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:37:36,173] [1217460] [ INFO] [ load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-2/conf/current' (rc=-2) — will fetch from primary after election
|
||||
[2026-06-21 08:37:36,173] [1217460] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=0)
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election
|
||||
[2026-06-21 08:37:36,673] [1217460] [ ERROR] [ cluster_init_phase2] cluster_init_phase2: cluster_config not loaded
|
||||
[2026-06-21 08:37:36,673] [1217460] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:46,177] [1217460] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:37:46,177] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:37:46,177] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8)
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:40:02,029] [1217460] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 585ms (priority=1) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:40:02,629] [1217460] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=1, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:40:03,270] [1217460] [ INFO] [ha_on_heartbeat_received] HA: primary heartbeat arrived mid-PreVote — cancelling the round and re-attaching as CTRL_STANDBY (leader alive, no term change) (Req 4.2)
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=9)
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ main] My name is ctrl-2
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
||||
[2026-06-21 08:46:07,149] [1227110] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=3)
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=22 current_lsn=23; applied_lsn=22 advertised for catch-up
|
||||
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:46:09,319] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:46:09,319] [1227110] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=6)
|
||||
[2026-06-21 08:47:38,988] [1227110] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 556ms (priority=1) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:39,276] [1227110] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:39,276] [1227110] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:39,277] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:39,277] [1227110] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=8)
|
||||
[2026-06-21 08:47:39,588] [1227110] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
|
||||
[2026-06-21 08:47:51,992] [1227110] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 662ms (priority=1) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:52,793] [1227110] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:47:52,840] [1227110] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4)
|
||||
[2026-06-21 08:47:52,843] [1227110] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9991)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:47:52,850] [1227110] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:47:52,850] [1227110] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed
|
||||
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=48 < target=49 → streaming [48+1..49]
|
||||
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=49
|
||||
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..49] to standby
|
||||
[2026-06-21 08:47:53,595] [1227110] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=49 < target=53 → streaming [49+1..53]
|
||||
[2026-06-21 08:47:53,595] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=50 target_lsn=53
|
||||
[2026-06-21 08:47:53,596] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [49+1..53] to standby
|
||||
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ main] My name is ctrl-2
|
||||
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
||||
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
||||
[2026-06-21 08:47:59,845] [1228672] [ INFO] [ cluster_init_phase1] HA: restored term=4 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
||||
[2026-06-21 08:47:59,854] [1228672] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=4)
|
||||
[2026-06-21 08:48:00,354] [1228672] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-2, voted_term=4, raft_term=4, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:48:00,354] [1228672] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up
|
||||
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
213
.kiro/specs/ha-commit-durability/repro-851/ctrl-3.log
Normal file
213
.kiro/specs/ha-commit-durability/repro-851/ctrl-3.log
Normal file
|
|
@ -0,0 +1,213 @@
|
|||
[2026-06-21 08:37:36,665] [1217472] [ INFO] [ main] My name is ctrl-3
|
||||
[2026-06-21 08:37:36,665] [1217472] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-3
|
||||
[2026-06-21 08:37:36,666] [1217472] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3)
|
||||
[2026-06-21 08:37:36,666] [1217472] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin
|
||||
[2026-06-21 08:37:36,672] [1217472] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:37:36,672] [1217472] [ INFO] [ load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-3/conf/current' (rc=-2) — will fetch from primary after election
|
||||
[2026-06-21 08:37:36,672] [1217472] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=0)
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election
|
||||
[2026-06-21 08:37:37,672] [1217472] [ ERROR] [ cluster_init_phase2] cluster_init_phase2: cluster_config not loaded
|
||||
[2026-06-21 08:37:37,672] [1217472] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:47,276] [1217472] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:37:47,276] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:37:47,276] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8)
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:40:01,930] [1217472] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1150ms (priority=2) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:40:02,636] [1217472] [ INFO] [ myreq_handler] HA: PreVote DENIED candidate='ctrl-2' term=1 reason=stale_term (my_term=0, floor tag=5298030106630493228 lsn=13)
|
||||
[2026-06-21 08:40:03,130] [1217472] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:40:03,176] [1217472] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3)
|
||||
[2026-06-21 08:40:03,176] [1217472] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:40:03,180] [1217472] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=13 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=14 advertised for catch-up
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=1)
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:40:03,229] [1217472] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:40:03,230] [1217472] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=14, writable_lsn gate armed
|
||||
[2026-06-21 08:40:06,333] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1)
|
||||
[2026-06-21 08:40:06,333] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:40:06,374] [1217472] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=15 (committed_tag=5298030106630493228 committed_lsn=14)
|
||||
[2026-06-21 08:40:33,348] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 10)
|
||||
[2026-06-21 08:40:33,348] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:41:03,364] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 20)
|
||||
[2026-06-21 08:41:03,364] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:41:33,382] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 30)
|
||||
[2026-06-21 08:41:33,382] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:42:03,397] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 40)
|
||||
[2026-06-21 08:42:03,397] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:42:33,411] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 50)
|
||||
[2026-06-21 08:42:33,411] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:43:03,425] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 60)
|
||||
[2026-06-21 08:43:03,425] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:43:33,240] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 70)
|
||||
[2026-06-21 08:43:33,240] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:43:45,247] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991)
|
||||
[2026-06-21 08:43:45,250] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent
|
||||
[2026-06-21 08:43:45,250] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 1, localhost:9991)
|
||||
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[0] lsn=13 < committed_lsn=22 → stream WAL batches
|
||||
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=14 target_lsn=22
|
||||
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [13+1..22] to standby
|
||||
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=14 < target=22 → streaming [14+1..22]
|
||||
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=15 target_lsn=22
|
||||
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [14+1..22] to standby
|
||||
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=15 < target=22 → streaming [15+1..22]
|
||||
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=16 target_lsn=22
|
||||
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [15+1..22] to standby
|
||||
[2026-06-21 08:43:48,849] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=16 < target=22 → streaming [16+1..22]
|
||||
[2026-06-21 08:43:48,849] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=17 target_lsn=22
|
||||
[2026-06-21 08:43:48,850] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [16+1..22] to standby
|
||||
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=17 < target=22 → streaming [17+1..22]
|
||||
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=18 target_lsn=22
|
||||
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [17+1..22] to standby
|
||||
[2026-06-21 08:43:49,250] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=18 < target=22 → streaming [18+1..22]
|
||||
[2026-06-21 08:43:49,250] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=19 target_lsn=22
|
||||
[2026-06-21 08:43:49,251] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [18+1..22] to standby
|
||||
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=19 < target=22 → streaming [19+1..22]
|
||||
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=20 target_lsn=22
|
||||
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [19+1..22] to standby
|
||||
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=20 < target=22 → streaming [20+1..22]
|
||||
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=21 target_lsn=22
|
||||
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [20+1..22] to standby
|
||||
[2026-06-21 08:43:49,851] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=21 < target=22 → streaming [21+1..22]
|
||||
[2026-06-21 08:43:49,851] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=22 target_lsn=22
|
||||
[2026-06-21 08:43:49,852] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [21+1..22] to standby
|
||||
[2026-06-21 08:43:51,294] [1217472] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=23 (committed_tag=5298030106630493228 committed_lsn=22)
|
||||
[2026-06-21 08:45:24,291] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1)
|
||||
[2026-06-21 08:45:24,292] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991)
|
||||
[2026-06-21 08:45:24,292] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 1)
|
||||
[2026-06-21 08:45:24,292] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111
|
||||
[2026-06-21 08:45:24,294] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent
|
||||
[2026-06-21 08:45:24,294] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 0, localhost:9991)
|
||||
[2026-06-21 08:45:51,305] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 10)
|
||||
[2026-06-21 08:45:51,305] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111
|
||||
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=35 < target=36 → streaming [35+1..36]
|
||||
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=36
|
||||
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..36] to standby
|
||||
[2026-06-21 08:46:09,314] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[1] (localhost:9992)
|
||||
[2026-06-21 08:46:09,319] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[1] established, ctrlhello sent
|
||||
[2026-06-21 08:46:09,319] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[1] (slot 2, localhost:9992)
|
||||
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[1] lsn=22 < committed_lsn=44 → stream WAL batches
|
||||
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=23 target_lsn=44
|
||||
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [22+1..44] to standby
|
||||
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=23 < target=44 → streaming [23+1..44]
|
||||
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=24 target_lsn=44
|
||||
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [23+1..44] to standby
|
||||
[2026-06-21 08:46:12,716] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=24 < target=44 → streaming [24+1..44]
|
||||
[2026-06-21 08:46:12,716] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=25 target_lsn=44
|
||||
[2026-06-21 08:46:12,717] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [24+1..44] to standby
|
||||
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=25 < target=44 → streaming [25+1..44]
|
||||
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=26 target_lsn=44
|
||||
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [25+1..44] to standby
|
||||
[2026-06-21 08:46:13,117] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=26 < target=44 → streaming [26+1..44]
|
||||
[2026-06-21 08:46:13,118] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=27 target_lsn=44
|
||||
[2026-06-21 08:46:13,118] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [26+1..44] to standby
|
||||
[2026-06-21 08:46:13,318] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=27 < target=44 → streaming [27+1..44]
|
||||
[2026-06-21 08:46:13,318] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=28 target_lsn=44
|
||||
[2026-06-21 08:46:13,319] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [27+1..44] to standby
|
||||
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=28 < target=44 → streaming [28+1..44]
|
||||
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=29 target_lsn=44
|
||||
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [28+1..44] to standby
|
||||
[2026-06-21 08:46:13,719] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=29 < target=44 → streaming [29+1..44]
|
||||
[2026-06-21 08:46:13,719] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=30 target_lsn=44
|
||||
[2026-06-21 08:46:13,720] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [29+1..44] to standby
|
||||
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=30 < target=44 → streaming [30+1..44]
|
||||
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=31 target_lsn=44
|
||||
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [30+1..44] to standby
|
||||
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=31 < target=44 → streaming [31+1..44]
|
||||
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=32 target_lsn=44
|
||||
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [31+1..44] to standby
|
||||
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=32 < target=44 → streaming [32+1..44]
|
||||
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=33 target_lsn=44
|
||||
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [32+1..44] to standby
|
||||
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=33 < target=44 → streaming [33+1..44]
|
||||
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=34 target_lsn=44
|
||||
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [33+1..44] to standby
|
||||
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=34 < target=44 → streaming [34+1..44]
|
||||
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=35 target_lsn=44
|
||||
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [34+1..44] to standby
|
||||
[2026-06-21 08:46:14,922] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=35 < target=44 → streaming [35+1..44]
|
||||
[2026-06-21 08:46:14,922] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=44
|
||||
[2026-06-21 08:46:14,923] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..44] to standby
|
||||
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=36 < target=44 → streaming [36+1..44]
|
||||
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=37 target_lsn=44
|
||||
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [36+1..44] to standby
|
||||
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=37 < target=44 → streaming [37+1..44]
|
||||
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=38 target_lsn=44
|
||||
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [37+1..44] to standby
|
||||
[2026-06-21 08:46:15,565] [1217472] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=45 (committed_tag=5298030106630493228 committed_lsn=44)
|
||||
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=38 < target=44 → streaming [38+1..44]
|
||||
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=39 target_lsn=44
|
||||
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [38+1..44] to standby
|
||||
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=39 < target=44 → streaming [39+1..44]
|
||||
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=40 target_lsn=44
|
||||
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [39+1..44] to standby
|
||||
[2026-06-21 08:46:16,124] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=40 < target=44 → streaming [40+1..44]
|
||||
[2026-06-21 08:46:16,124] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=41 target_lsn=44
|
||||
[2026-06-21 08:46:16,125] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [40+1..44] to standby
|
||||
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=41 < target=44 → streaming [41+1..44]
|
||||
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=42 target_lsn=44
|
||||
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [41+1..44] to standby
|
||||
[2026-06-21 08:46:16,525] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=42 < target=44 → streaming [42+1..44]
|
||||
[2026-06-21 08:46:16,525] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=43 target_lsn=44
|
||||
[2026-06-21 08:46:16,526] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [42+1..44] to standby
|
||||
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=43 < target=44 → streaming [43+1..44]
|
||||
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=44 target_lsn=44
|
||||
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [43+1..44] to standby
|
||||
[2026-06-21 08:47:29,557] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=45 < target=48 → streaming [45+1..48]
|
||||
[2026-06-21 08:47:29,557] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=46 target_lsn=48
|
||||
[2026-06-21 08:47:29,558] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [45+1..48] to standby
|
||||
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=51 → streaming [46+1..51]
|
||||
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=51
|
||||
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..51] to standby
|
||||
[2026-06-21 08:47:36,064] [1228337] [ INFO] [ main] My name is ctrl-3
|
||||
[2026-06-21 08:47:36,064] [1228337] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-3
|
||||
[2026-06-21 08:47:36,065] [1228337] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3)
|
||||
[2026-06-21 08:47:36,065] [1228337] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin
|
||||
[2026-06-21 08:47:36,071] [1228337] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=3)
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=52 current_lsn=53; applied_lsn=52 advertised for catch-up
|
||||
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:47:52,078] [1228337] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1088ms (priority=2) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:47:53,279] [1228337] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
|
||||
[2026-06-21 08:48:04,083] [1228337] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1092ms (priority=2) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:48:05,284] [1228337] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
1 | 0 | NOOP | 0 | 00000000 |
|
||||
2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
14 | 0 | NOOP | 0 | 00000000 |
|
||||
15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
45 | 0 | NOOP | 0 | 00000000 |
|
||||
46 | 0 | JOB_SUBMIT | 3952 | faead246 | jobid=1868595630964736 subtime=1782031661 state=1
|
||||
47 | 0 | JOB_SUBMIT | 3952 | faead246 | jobid=1868595630964736 subtime=1782031661 state=1
|
||||
48 | 0 | JOB_SUBMIT | 3952 | be0059bf | jobid=1868595630964737 subtime=1782031661 state=1
|
||||
49 | 0 | JOB_SUBMIT | 3952 | be0059bf | jobid=1868595630964737 subtime=1782031661 state=1
|
||||
50 | 0 | JOB_SUBMIT | 3952 | fe7a3d0b | jobid=1868595630964738 subtime=1782031661 state=1
|
||||
51 | 0 | JOB_SUBMIT | 3952 | fe7a3d0b | jobid=1868595630964738 subtime=1782031661 state=1
|
||||
52 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
|
||||
53 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
|
||||
54 | 0 | NOOP | 0 | 00000000 |
|
||||
segments: 3 entries: 54 submits: 50
|
||||
lsn range: 1..54 recovered tip: 54
|
||||
gaps: 0 crc chain: intact
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
1 | 0 | NOOP | 0 | 00000000 |
|
||||
2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
14 | 0 | NOOP | 0 | 00000000 |
|
||||
15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
45 | 0 | NOOP | 0 | 00000000 |
|
||||
46 | 0 | JOB_SUBMIT | 3952 | d32ac747 | jobid=1868595643547648 subtime=1782031673 state=1
|
||||
47 | 0 | JOB_SUBMIT | 3952 | d32ac747 | jobid=1868595643547648 subtime=1782031673 state=1
|
||||
48 | 0 | JOB_SUBMIT | 3952 | 97c04cbe | jobid=1868595643547649 subtime=1782031673 state=1
|
||||
49 | 0 | JOB_SUBMIT | 3952 | 97c04cbe | jobid=1868595643547649 subtime=1782031673 state=1
|
||||
50 | 0 | JOB_SUBMIT | 3952 | d7ba280a | jobid=1868595643547650 subtime=1782031673 state=1
|
||||
51 | 0 | JOB_SUBMIT | 3952 | d7ba280a | jobid=1868595643547650 subtime=1782031673 state=1
|
||||
52 | 0 | JOB_SUBMIT | 3952 | 1e155b4c | jobid=1868595643547651 subtime=1782031673 state=1
|
||||
53 | 0 | JOB_SUBMIT | 3952 | 1e155b4c | jobid=1868595643547651 subtime=1782031673 state=1
|
||||
54 | 0 | NOOP | 0 | 00000000 |
|
||||
segments: 3 entries: 54 submits: 50
|
||||
lsn range: 1..54 recovered tip: 54
|
||||
gaps: 0 crc chain: intact
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
1 | 0 | NOOP | 0 | 00000000 |
|
||||
2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
14 | 0 | NOOP | 0 | 00000000 |
|
||||
15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
45 | 0 | JOB_SUBMIT | 3952 | 4c6d24e6 | jobid=1868595618381824 subtime=1782031649 state=1
|
||||
46 | 0 | JOB_SUBMIT | 3952 | 4c6d24e6 | jobid=1868595618381824 subtime=1782031649 state=1
|
||||
47 | 0 | JOB_SUBMIT | 3952 | 0887af1f | jobid=1868595618381825 subtime=1782031649 state=1
|
||||
48 | 0 | JOB_SUBMIT | 3952 | 0887af1f | jobid=1868595618381825 subtime=1782031649 state=1
|
||||
49 | 0 | JOB_SUBMIT | 3952 | 48fdcbab | jobid=1868595618381826 subtime=1782031649 state=1
|
||||
50 | 0 | JOB_SUBMIT | 3952 | 48fdcbab | jobid=1868595618381826 subtime=1782031649 state=1
|
||||
51 | 0 | JOB_SUBMIT | 3952 | 8152b8ed | jobid=1868595618381827 subtime=1782031649 state=1
|
||||
52 | 0 | JOB_SUBMIT | 3952 | 8152b8ed | jobid=1868595618381827 subtime=1782031649 state=1
|
||||
53 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
|
||||
54 | 0 | NOOP | 0 | 00000000 |
|
||||
segments: 2 entries: 54 submits: 51
|
||||
lsn range: 1..54 recovered tip: 54
|
||||
gaps: 0 crc chain: intact
|
||||
|
|
@ -36,3 +36,63 @@ Date: 2026-06-11. Discovery status: **complete via umbrella** — this sub-spec
|
|||
- **Standalone behavior change (intentional)**: with the engine wired, `n_peers <= 1` sync appends now advance `committed_lsn` (previously the -ENODEV path left it unmoved). Raft-correct (self-quorum), satisfies spec req 6.3; flagged in design.
|
||||
- **Membership-shrink semantics**: re-evaluation = wake + re-check; no forced early-fail (umbrella 4.3 requires fail-not-confirm, satisfied by timeout; reconnection within the deadline may still legitimately commit).
|
||||
- Review gate: design verified against the umbrella contract — interface verbatim, no contradiction with locked decisions, all 20 slice requirement IDs traced.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
**Type:** Post-implementation gap audit (not a pre-impl gap analysis). Roadmap declares `ha-commit-engine` IMPLEMENTED 12/12 (2026-06-12, validate-impl GO, 467 tests green). `tasks.md` shows 6 `[ ]` checkboxes; all 6 are **section/group headers** ("1.", "2.", "3.", "4.", "5.", "6."), not leaf tasks — every leaf sub-task (1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1) is `[x]`. The "12/12" count is the leaf count. The unchecked group headers are an artifact of the checklist format, documented as such in §Implementation Notes — they are **not** deferred work and **not** core gaps.
|
||||
|
||||
### Verdict
|
||||
|
||||
**STATUS: CLOSED.** All 7 requirements (20 EARS acceptance criteria) are implemented and wired in production. Verified against current on-disk source (codegraph verbatim reads + grep, 2026-06-18). The engine has since been extended by `ha-commit-durability` (durable self-cap) — that extension strengthens, and does not regress, the ha-commit-engine contract.
|
||||
|
||||
### Requirement-to-Asset Map (verified, file:line)
|
||||
|
||||
| Req | Criterion | Asset (verified) | Status |
|
||||
|-----|-----------|------------------|--------|
|
||||
| 1.1 | confirm only after per-entry majority | `commit_submit(WAIT)` predicate `wait_done_locked` commit.c:184-189; sync path persist.c:1897-1900 → `repl_wait_majority_ack` walAck delegation repl.c:447 → `commit_submit` | MET |
|
||||
| 1.2 | count ACK toward N only if acked_lsn >= N | `majority_candidate_locked` reads `s->acked_lsn`, sorts desc, candidate = `scratch[n_peers/2-1]` commit.c:107-134 | MET |
|
||||
| 1.3 | connectivity never substitutes for ACKs | ACK-time eval only; watermarks read from registry at `commit_on_ack` commit.c:366-381; gate (ha_has_write_quorum) is fast-reject only, engine re-verifies | MET |
|
||||
| 2.1 | no commit advance without majority | `commit_advance_locked` stores only `majority_candidate_locked` result commit.c:145-168 | MET |
|
||||
| 2.2 | monotonic committed_lsn | `if (cand <= cur) return;` + `atomic_store(max)` commit.c:150-165; XCE_TESTING monotonic assert | MET |
|
||||
| 2.3 | contiguous, bounded by appended log | candidate clamped to `cap` (durable, ≤ `highest_appended_lsn`) commit.c:136-137, 52-55; cumulative watermarks → contiguous | MET (strengthened: cap is now durable_lsn per ha-commit-durability) |
|
||||
| 3.1/3.2/3.3 | admission quorum counts only members able to receive; catch-up + disconnected excluded | `repl_active_standby_count` (ep != NULL && !in_catch_up) repl.c:668-686; consumed by `ha_has_write_quorum` ha.c:2049-2058 | MET |
|
||||
| 4.1 | post-acceptance shrink → fail not confirm | WAIT times out (`-ETIMEDOUT`) when majority never reached; no false-success path; `commit_on_membership_change` re-evals monotonically (shrink never regresses) commit.c:397-409 | MET |
|
||||
| 4.2 | membership change re-evaluates pending waits | `commit_on_membership_change` wired at 4 mutators: repl.c:130 (add), :162 (remove), :598 (start_catchup), :621 (complete_catchup) | MET |
|
||||
| 4.3/4.4 | zero peer ACKs never majority (n_peers>1); no zero-active short-circuit | engine derives from `n_peers` not n_active; `k = n_peers/2-1`, absent peers zero-filled commit.c:117-134; legacy `n_active==0→success` short-circuit removed (sync path persist.c:1897 engine branch; configAck has explicit "no n_active shortcut" repl.c:506-510) | MET |
|
||||
| 5.1 | lag never blocks majority | majority is the only threshold; far-lagging standby sorts low, does not affect `scratch[n_peers/2-1]` commit.c:129-134 | MET |
|
||||
| 6.1 | reject without quorum | `ha_has_write_quorum` false → `ha_dispatch_gate` -XCE_ERR_NOT_READY ha.c:2031-2059 | MET |
|
||||
| 6.2 | auto-resume on quorum restore | gate is stateless per request (recomputes `repl_active_standby_count` each call); no reset needed | MET |
|
||||
| 6.3 | standalone self-quorum | `n_peers <= 1` → `commit_advance_locked` immediate, returns cap commit.c:100-102, 330-332; sync path -ENODEV special case removed on engine branch | MET |
|
||||
| 7.1/7.2 | n_vcpus-relative admission cap, NOT_READY beyond | `wait_cap = max(1, n_vcpus - COMMIT_WAIT_RESERVED)` commit.c:175-180; `inflight_waits >= cap → -XCE_ERR_NOT_READY` commit.c:313-320 | MET |
|
||||
| 7.3 | ACK processing unaffected at cap | `commit_on_ack` takes only ce->lock, never the inflight gate; advances + broadcasts regardless of cap commit.c:366-381 | MET |
|
||||
|
||||
### The 6 Open `[ ]` Tasks — Classified
|
||||
|
||||
All 6 are group-header checkboxes whose every child is `[x]`. None is deferred work; none is a core gap. They are a **checklist-rendering artifact** (parent headers left unchecked while leaves complete).
|
||||
|
||||
| Header | Children | Classification | Effort/Risk |
|
||||
|--------|----------|----------------|-------------|
|
||||
| 1. Commit engine core | 1.1 [x], 1.2 [x] | DONE (artifact) | none |
|
||||
| 2. WAIT/self-quorum/zero-peer/cap/gate | 2.1 [x], 2.2 [x] | DONE (artifact) | none |
|
||||
| 3. Membership re-evaluation | 3.1 [x], 3.2 [x] | DONE (artifact; jointly completed — RED unachievable, §Notes) | none |
|
||||
| 4. Quorum-accounting fix | 4.1 [x], 4.2 [x] | DONE (artifact) | none |
|
||||
| 5. Integration (wire engine behind job submit) | 5.1 [x], 5.2 [x], 5.3 [x] | DONE (artifact) | none |
|
||||
| 6. Validation / dishonest-path audit | 6.1 [x] | DONE (artifact; 6.1 audit record in §Notes) | none |
|
||||
|
||||
**Bottom line: 0 deferred follow-ups remain as core gaps. The "deferred follow-ups in §Implementation Notes" referenced by the roadmap are advisory/out-of-scope items, not open tasks** — see the genuine residuals below; all are explicitly out-of-spec-boundary and owned elsewhere.
|
||||
|
||||
### Genuine Residuals (advisory, all out-of-boundary or already resolved downstream)
|
||||
|
||||
These are the §Implementation Notes "deferred follow-ups". None blocks CLOSED status; each is owned by a later spec or is cosmetic.
|
||||
|
||||
1. **LSN-0 first-entry legacy edge** (Notes 5.1, 6.1 audit). The first-ever WAL entry (LSN 0) routes to the legacy branch (persist.c:1903-1929) because `commit_submit` rejects `lsn==0`. — **RESOLVED downstream:** ha-single-leader-role's promotion no-op entry makes LSN 0 never a client write in practice; repl.c:438-440 now confirms LSN 0 immediately. Once-per-cluster-lifetime, no consistency loss. *Effort: 0 (closed). Risk: none.*
|
||||
2. **`ce->n_peers` not re-derived on dynamic membership change** (Notes 6.1(b)). If `ha->n_peers` is reassigned post-init (e.g. ha.c RDHUP re-election), the engine's `n_peers` is stale. — **Out of boundary** (dynamic membership = future work); current cluster topology is fixed at init. *Option A: assert n_peers stability at re-election callback; Option B: re-derive `ce->n_peers` in a `commit_set_n_peers` setter called from the election path; Option C: leave as documented latent gap. Effort: A ~0.5d, B ~1d (+revalidate barrier-rollout/durability). Risk: LOW today (no dynamic membership in production); MEDIUM if dynamic reconfiguration is ever added without addressing it.*
|
||||
3. **Pre-existing ASAN leaks in 6 test groups** (Notes 6.1 LEAK FOLLOW-UP: worker_join 33KB, rdhup_integration, failover, start_election_async, liveness, connection_reuse). — **Out of scope** (pre-existing, production-side allocations; suite runs with `detect_leaks=0`). *Option: dedicated leak-cleanup pass. Effort: ~1-2d. Risk: LOW (test-harness hygiene, not a correctness gap).*
|
||||
4. **Stale comment persist.c:1151-1154 / pre-engine contract text** (Notes 6.1). One-line doc follow-up. *Effort: trivial. Risk: none.*
|
||||
5. **Untracked `src/lib/testsuite/test_wal_wire.c`** referenced by committed meson.build (Notes 5.1). Repo hygiene. *Effort: trivial (git add). Risk: none.*
|
||||
|
||||
### Drift Note (informational)
|
||||
|
||||
The engine source has been extended by **ha-commit-durability** (implemented 2026-06-18) AFTER ha-commit-engine: `majority_candidate_locked` now caps the self-contribution at `persist_durable_lsn` (commit.c:93) rather than the appended tip, and a `commit_on_durable_advance` flusher wake was added (commit.c:218). This is additive — it tightens Req 2.3 (committed never beyond durable, not merely never beyond appended) and closes bug-661. The ha-commit-engine contract is unchanged and remains fully satisfied. Original research-table anchors (e.g. repl.c:432, persist.c:1035, ha.c:1507) have drifted; verified-current anchors are in the Requirement-to-Asset Map above.
|
||||
|
|
|
|||
|
|
@ -710,3 +710,85 @@ Aggregate remaining: **~M total for Phase 1 (all S items), ~L for Phase 2** (G2+
|
|||
- **RN3**: Restart-path divergence (G5) — construct the failure: standby with uncommitted WAL tail restarts, rejoins, receives catch-up; verify whether `apply_wal_entry` overwrite semantics mask or corrupt the divergent tail.
|
||||
- **RN4**: All-zero cold-start convergence time and liveness (G6) — 3-node cluster, no configs anywhere; confirm watchdog-driven election converges and measure time-to-primary.
|
||||
- **RN5**: Serve-time enforcement for 8.4 — can a promoted standby serve a config tag above `committed_config_tag` in any window between promotion and first quorum push?
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
> Trigger: post-implementation gap audit. Tasks 46/46 `[x]`, `phase=implemented`.
|
||||
> Method: re-verified the 2026-06-09 gap list (G1–G10) against the current working
|
||||
> tree on `persistent_ha`. Since the prior pass, two successor specs landed and
|
||||
> closed several of the gaps this spec recorded: **`libxce-failover`** (client-side
|
||||
> transparent failover/redirect) and **`ha-commit-durability`** (bug-661, landed
|
||||
> 2026-06-18). The earlier majority-commit model was also superseded by a
|
||||
> `wal_classify()` WAIT/RETAIN scheme from `ha-commit-barrier-rollout`. Line numbers
|
||||
> below reflect the current tree (ha.c is now 5,338 lines; persist.c 2,587).
|
||||
|
||||
### Headline change vs 2026-06-09
|
||||
|
||||
The 2026-06-09 pass found ~40/50 ACs fully implemented with 10 open gaps (G1–G10).
|
||||
**Five of those gaps are now CLOSED** by successor-spec work; the residual set is
|
||||
smaller and almost entirely *spec-text reconciliation debt* rather than missing code.
|
||||
|
||||
### Requirement-to-Asset Map — Delta Since 2026-06-09
|
||||
|
||||
| Gap (2026-06-09) | Req | New Status (2026-06-18) | Evidence |
|
||||
|---|---|---|---|
|
||||
| **G1** Redirect payload never populated | R1.4, R7.2 | **CLOSED** | `fill_redirect_reply()` populates `reply->redirect_host/redirect_port` from `ha->primary_host/port` (`join.c:28-32`); called on the `-E_XCE_REDIRECT` path (`join.c:282`); covered by `test_join_redirect.c`. Comment cites `libxce-failover Req 8.1`. |
|
||||
| **G7** libxce retry has zero adopters | R7 obj. | **CLOSED (re-homed)** | Orphaned no-op `xce_request_with_retry()` + `xce_retry_send_hook` were **removed** (`request.c:1923-1936`); superseded by the `xcecli_*` failover loop in `src/lib/client.c` + `src/lib/client_failover.c`. `XCE_RETRY_DEFAULTS` retained as the shared tunable. Production redirect-follow is now live end-to-end (G1 + client_failover). |
|
||||
| **G10** `ha_find_and_connect_to_primary` dead code | cleanup | **CLOSED** | No production `.c` reference remains (grep over `src/controller/*.c` returns nothing). Dead-code deletion done. |
|
||||
| **G2** WAL majority-commit covers only job submit | R5.1 | **SUPERSEDED (model changed)** | Replaced by `wal_classify()` WAIT/RETAIN scheme (`persist.c:2004`). `WAL_JOB_SUBMIT` and `WAL_QUEUE_CTRL` are `WAL_MODE_WAIT` (commit-gated before requester reply); `WAL_WORKER_CTRL` JOIN/UPDATE are WAIT; job state transitions (`WAL_JOB_START/FINISH/FAIL/...`) are `WAL_MODE_RETAIN` — replicated and retained-until-committed but not blocking the worker-fact report. The request-shaped side of dual-origin events is realized by the handler barrier in `do_jobcontrol_op`. The original "everything sync-blocks" framing was deliberately not adopted (latency); coverage is now classification-driven, owned by `ha-commit-barrier-rollout` / `ha-commit-durability`. |
|
||||
| **G3** Commit timeout degraded-mode vs CP | R5.5 | **SUPERSEDED (bug-661 / ha-commit-durability)** | Durability invariant now GUARANTEED: acknowledged ⇒ `fdatasync`'d on a fault-tolerant majority. `majority_candidate_locked` caps the primary self-contribution at `persist_durable_lsn`; `fsync_worker` group-commit flusher advances `durable_lsn` only on `fdatasync` success (fail-loud rc-gate); standby `fdatasync`s before WalAck. Verified by `test_commit_durability.c`, the ack-before-fdatasync E2E, and the rolling-failover soak (zero acked-job loss). Controller CLAUDE.md checklist item flipped to GUARANTEED 2026-06-18. The old "silent local-only degrade" path is gone. |
|
||||
|
||||
### Residual Gaps (still open after 2026-06-18)
|
||||
|
||||
| ID | Req | Status | Detail |
|
||||
|---|---|---|---|
|
||||
| **R1 (was G6)** Config probe path unwired | R3.1, R3.3, R3.4 | **Missing-as-specified (stable)** | `xce_probe_config_tag()` is fully implemented + unit-tested (`config_repl.c:1219`) but **still not called in production** — `myapp.c` has no probe call (grep confirms). The tag-0 voter-only gate (`ha.c:4495`) uses static `my_config_tag`/`n_peers`, not probe results. The comment at `ha.c:4493` ("xce_probe_config_tag path already called in myapp_run") is **stale/incorrect** — no such call exists. All-zero cold-start converges only via watchdog-driven re-election (~heartbeat_timeout), and that path lacks an automated convergence test (prior RN4). |
|
||||
| **R2 (was G9)** Spec/code reconciliation debt | 2.8, 2.10–2.12, 3.1–3.5, 5.5, 6.1 | **Open (docs only)** | requirements.md EARS text still describes: fatal-exit for tag-0 candidate (code: voter-only standby), exactly-once split-vote retry (code: unbounded watchdog retry loop), exponential-backoff probe (code: passive watchdog), standby-dials-primary (code: primary dials out — inverted by design per controller CLAUDE.md), CP-timeout-error (code: now GUARANTEED durability, but worded as the old model). spec.json `note` still lists the now-closed G1/G3 framings. The TLA+ mapping + controller CLAUDE.md are authoritative and correct; the spec is the lagging artifact. |
|
||||
| **R3 (was part of G4)** Single-node `committed_lsn` runtime advance | R5.8 | **Likely closed — verify** | `test_single_node_committed_lsn_advances` (`test_p2_integration.c:251`) and `test_legacy_sync_append_holds_committed_at_durable` (`test_commit_durability.c:660`) now exist, suggesting ha-commit-durability addressed the prior `-ENODEV` short-circuit concern. Recommend a quick read of those tests to confirm RN1 is retired before closing. |
|
||||
| **R4 (was G5)** Standby restart-path divergent-tail truncation | R5.7 | **Re-scoped to recovery** | The RDHUP/watchdog failover path truncates to `last_known_committed_lsn`. The *restart* path is now governed by `ha-commit-durability` recovery (`recover_verdict_t` clean-vs-corruption, durable-floor re-derivation, pre-promotion fence). Whether a restarted standby with an uncommitted tail is correctly fenced/caught-up before rejoin is owned by that spec's recovery tests, not this one. Verify via `ha-commit-durability` rather than re-opening here. |
|
||||
| **R5 (was G8)** Consolidated election-outcome log | R10.1 | **Open (minor)** | FSM-transition INFO logs carry term, and majority-ack logs carry acks+quorum, but there is still no single election-complete record emitting *votes-granted AND quorum-size* together. Data is available in `ha_elect_ctx_t`/raft state at the decision point. Low value, low effort. |
|
||||
|
||||
### Residual Tally
|
||||
|
||||
- **Closed since 2026-06-09:** G1, G7, G10 (outright), G2, G3 (superseded by a better model in successor specs).
|
||||
- **Still open:** R1 (probe unwired + stale comment), R2 (spec-text reconciliation), R5 (consolidated election log). R3 likely-closed-pending-verify, R4 re-homed to ha-commit-durability recovery.
|
||||
- **Net AC posture:** the durability and redirect deltas mean the *functional* CP promise of this spec is now met or exceeded; the open items are documentation honesty (R2), one unwired-but-replaced bootstrap path (R1), and one cosmetic log (R5).
|
||||
|
||||
### Implementation Approach Options (for the residual work)
|
||||
|
||||
#### Option A — Close-and-reconcile in place (recommended)
|
||||
Phase-1-only, all S-size, no successor spec needed:
|
||||
1. **R2** rewrite requirements.md AC 2.8 / 2.10–2.12 / 3.1–3.5 / 5.5 / 6.1 to describe the as-built push/watchdog/outbound-primary/guaranteed-durability architecture; refresh the spec.json `note` (G1/G3 are no longer "known deviations"). Docs only.
|
||||
2. **R1** delete the stale `ha.c:4493` comment; either (a) delete the unused `xce_probe_config_tag` production path and document the static-gate + watchdog-convergence model, or (b) add a cold-start convergence integration test (3-node, all configs absent) so the watchdog fallback is covered. Recommend (a)+test.
|
||||
3. **R5** add a single INFO "election complete: term=T votes=G/Q role=…" record at the become-leader / become-follower decision point.
|
||||
4. **R3** read the two new single-node durability tests; if they cover runtime advance, retire RN1 and mark R5.8 closed.
|
||||
- ✅ Closes the spec honestly with no policy decisions left implicit; smallest surface.
|
||||
- ✅ Every item has an existing seam + testsuite file.
|
||||
- ❌ Touches requirements.md (aging artifact) — but that is the point.
|
||||
- **Effort: S total. Risk: Low** (docs + dead-code delete + one log line + one test).
|
||||
|
||||
#### Option B — Defer reconciliation, close only code defects
|
||||
Fix R1 comment + R5 log now; leave requirements.md as-is with an expanded spec.json deviation note pointing at controller CLAUDE.md as the source of truth.
|
||||
- ✅ Even smaller; no requirements.md churn.
|
||||
- ❌ Leaves EARS text contradicting shipped behavior — the recurring complaint across every prior pass. Reconciliation debt compounds.
|
||||
- **Effort: XS. Risk: Low (but accrues doc debt).**
|
||||
|
||||
#### Option C — Fold residuals into successor specs and close this one verbatim
|
||||
Mark ha-config-bootstrap closed; track R1/R2/R5 under `ha-commit-durability` (R3/R4 already there) or a small `ha-spec-reconcile` doc spec.
|
||||
- ✅ This spec stops being touched; clean lineage.
|
||||
- ❌ R2 spans this spec's own EARS text — odd to reconcile it from another spec. R5 is trivially this spec's.
|
||||
- **Effort: S (mostly bookkeeping). Risk: Low; tracking-loss risk medium.**
|
||||
|
||||
### Effort & Risk — Residual
|
||||
|
||||
| Item | Effort | Risk | Notes |
|
||||
|---|---|---|---|
|
||||
| R1 probe path + stale comment + cold-start test | S | Low | Behavior already converges via watchdog; delete dead path or add test |
|
||||
| R2 requirements.md / spec.json reconciliation | S | Low | Docs only; 6 EARS deltas; CLAUDE.md is the correct source |
|
||||
| R3 single-node commit verify | XS | Low | Read 2 existing tests; likely already closed |
|
||||
| R4 restart-path truncation | — | — | Re-homed to ha-commit-durability recovery; verify there, not here |
|
||||
| R5 consolidated election log | S | Low | One INFO line; data already in elect_ctx |
|
||||
|
||||
**Recommendation:** Option A. The heavy lifting (durability, redirect, retry adoption, dead-code) is done by successor specs; the only thing standing between this spec and an honest `implemented`/closed state is requirements-text reconciliation (R2), one unwired-but-superseded path (R1, delete + document), and a cosmetic log (R5). All S/XS, all Low risk.
|
||||
|
|
|
|||
|
|
@ -236,6 +236,13 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
|
|||
- _Depends: 9.2_
|
||||
|
||||
- [x] 10. Change G: libxce client-side transparent retry (P)
|
||||
- **NOTE (2026-06-16):** only the standalone primitive (10.2 `xce_request_with_retry()`) +
|
||||
its unit tests (10.1) were delivered. The primitive is **not wired into any production
|
||||
request path** — `xce_*_req` wrappers still call plain `xce_request()` — and it lacks
|
||||
multi-master rotation to discover a new primary after the connected one is killed (it only
|
||||
follows an explicit redirect). Production integration + master-list failover are owned by
|
||||
the **`libxce-failover`** spec (see `.kiro/specs/libxce-failover/brief.md`). Req 7 is
|
||||
therefore satisfied at the primitive level only; transparent failover is not yet live.
|
||||
- [x] 10.1 Write failing tests for xce_request_with_retry()
|
||||
- In `src/lib/testsuite/test_libxce_retry.c`: test `XCE_ERR_NOT_READY` → backoff + retry same host; test `XCE_ERR_STANDBY` with `redirect_host/port` → reconnect to redirect + retry; test `XCE_ERR_STANDBY` without redirect → fallback to backoff; test `XCE_ERR_AUTH` → surface immediately (no retry); test `max_retries` exhausted → surface last error; test 50 concurrent simulated clients → all succeed within 30s (backoff prevents starvation)
|
||||
- `make test` fails
|
||||
|
|
|
|||
|
|
@ -166,3 +166,103 @@ The five gaps:
|
|||
5. **No conflict detection in standby apply**: `apply_wal_entry` advances `current_lsn` and applies unconditionally by type (persist.c:720-721) — entries from the new primary are applied on top of phantom state with no divergence check.
|
||||
|
||||
Eviction strategy options for design: (i) in-memory rebuild on demotion — new teardown for jobmgr/queues/workers, then truncate + `replay_wal` from checkpoint (most work, no restart); (ii) process self-restart on demotion after truncation — minimal code, CP-acceptable since the node is leaving leadership anyway; (iii) entity-bearing snapshot transfer from the new primary — depends on deferred Task 11 snapshot format. **DECIDED (user, 2026-06-11): option (ii) — process self-restart on demotion** (after truncating the divergent suffix). Simplest and battle-tested: recovery reuses the boot path; a demoted node's worker/scheduler connections migrate to the new primary anyway, so the restart's connection reset costs little. Rebuild-in-place (i) remains a future refinement only if restart proves operationally disruptive. For gap 4, note Raft treats commitIndex as VOLATILE (followers recover it from the leader): the Raft-conformant fix is leader-driven log reconciliation at rejoin (consistency check + truncate divergent suffix, then rebuild) rather than persisting `committed_lsn` locally — design should weigh reconciliation-at-rejoin vs local persistence; either must ensure the replayed-but-uncommitted suffix cannot leak through irrevocable egress before reconciliation completes.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
Roll-up gap audit of the UMBRELLA contract against the landed sub-specs. The umbrella
|
||||
(`ha-consistency-invariants`) is design-generated and owns NO tasks of its own; it was
|
||||
decomposed (2026-06-11) into five implementing sub-specs. This pass maps every umbrella
|
||||
acceptance criterion to its owning sub-spec and verifies it against real code (file:line),
|
||||
then enumerates the invariants still OPEN with their owning future specs.
|
||||
|
||||
**Method**: codegraph + grep against `src/controller/{commit.c,persist.c,ha.c,ha_convergence.c,config_repl.c,egress.c,job.c}`; sub-spec `tasks.md` checkbox counts and `spec.json`; roadmap backlog (`.kiro/steering/roadmap.md`) for future-spec ownership.
|
||||
|
||||
### Sub-spec landing status (verified against code, not just spec.json phase)
|
||||
|
||||
> `spec.json.phase` is STALE for three sub-specs (still reads `tasks-generated`) and several
|
||||
> `tasks.md` checkboxes were never re-ticked. The authoritative signal is the on-disk code +
|
||||
> git history: the COMMITTING machinery is **fully gone**, the commit engine is live, demotion
|
||||
> convergence is wired, and `ha-commit-durability` (23/23 tasks, all committed d9b43c2…1ca2d64)
|
||||
> re-verified the whole protocol end-to-end. The audit below grades by CODE, flagging the
|
||||
> stale-tracking discrepancy where it occurs.
|
||||
|
||||
| Sub-spec | tasks.md ticks | Code verdict | Notes |
|
||||
|---|---|---|---|
|
||||
| ha-commit-engine | 12/18 ticked | **LANDED** | `commit.c` present & called from persist/ha/repl; per-LSN eval, self-quorum, own-term gate, admission cap all in code. Open boxes are stale tracking. |
|
||||
| ha-commit-barrier-rollout | 23/23 | **LANDED** | `wal_classify` totality (persist.c:2004) + idempotency guards (job.c:627-797) + reply-after-commit. |
|
||||
| ha-single-leader-role | 11/17 ticked | **LANDED** | Zero `CTRL_PRIMARY_COMMITTING`/`committing_timeout_fn`/`ha_check_majority_convergence`/`repl_ack_count` refs in src/controller (excl. testsuite). Open boxes are stale tracking. |
|
||||
| ha-convergence-recovery | 18/19 | **LANDED (1 E2E open)** | `ha_convergence.c` present + demotion path wired (ha.c:1188-1232); egress filter (egress.c). Only open: task 8 = live 3-node E2E test. |
|
||||
| ha-commit-durability | 23/23 | **LANDED** | Group-commit flusher + `durable_lsn`, commit≤durable invariant, durable-backed vote freshness, no-self-terminate fence. |
|
||||
|
||||
### Requirement → sub-spec → code roll-up map
|
||||
|
||||
| Umbrella criterion | Owning sub-spec | Code evidence (file:line) | Verdict |
|
||||
|---|---|---|---|
|
||||
| 1.1, 1.2 direct PRIMARY, single role | single-leader-role | `ha_promote_to_primary` ha.c:2298; no COMMITTING enum anywhere | MET |
|
||||
| 1.3 no redirect from primary | single-leader-role | `ha_should_redirect` ha.c:2187 (standby-only redirect) | MET |
|
||||
| 1.4 status reports primary pre-own-term | single-leader-role | role set CTRL_PRIMARY directly at promotion (ha.c:2298-2315) | MET |
|
||||
| 1.5 / 3.3 no wall-clock leadership change | single-leader-role | `committing_timeout_fn` deleted (zero refs); `ha_check_priority_stepdown` deleted (ha.c:1472) | MET |
|
||||
| 2.1 confirm only after majority ACK | commit-engine + barrier-rollout | `commit_submit` WAIT commit.c:284; reply-after-commit per classification | MET |
|
||||
| 2.2 requester ops fail on commit timeout | commit-engine | `-ETIMEDOUT` from `commit_submit` commit.c:340-359 | MET |
|
||||
| 2.3 no commit-index advance without majority | commit-engine | `commit_advance_locked` (median of acked_lsn); per-LSN eval | MET |
|
||||
| 2.4 provisional state, irrevocable egress gated | convergence-recovery | `job_submit_committed`/`collect_jobs_committed` egress.c:47-109 | MET |
|
||||
| 2.5 single-outcome convergence incl. memory | convergence-recovery | demotion truncate-above-committed + self-restart ha.c:1188-1232, ha_convergence.c | MET |
|
||||
| 2.6 monotonic commit index | commit-engine | monotonic max store + XCE_TESTING assert (commit.c) | MET |
|
||||
| 2.7 reconstruct from committed log | convergence-recovery + durability | boot recovery replay; `recovery_truncated` verdict (durability task 3.1) | MET |
|
||||
| 2.8 facts retained+retried until committed | commit-engine + barrier-rollout | RETAIN mode commit.c:296-302; log-is-the-queue; idempotency guards job.c:627-797 | MET |
|
||||
| 3.1 no confirm before own-term entry commits | single-leader-role + commit-engine | `ha_arm_own_term_gate` ha.c:2259 + `commit_set_writable_lsn` commit.c:383 (`writable_lsn` gate) | MET |
|
||||
| 3.2 pre-own-term: barrier or NOT_READY | commit-engine | own-term gate withholds WAIT completion; admission cap → NOT_READY | MET |
|
||||
| 4.1 quorum counts only members able to receive | commit-engine | `in_catch_up` exclusion (ha.c:325; commit eval excludes catch-up) | MET |
|
||||
| 4.2 commit requires per-entry ACKs | commit-engine | `commit_on_ack` reads registry `acked_lsn`, per-LSN median commit.c:366 | MET |
|
||||
| 4.3 post-acceptance quorum shrink → fail | commit-engine | ACK-time re-eval + WAIT timeout; `commit_on_membership_change` commit.c:397 | MET |
|
||||
| 4.4 never zero-peer majority (n_peers>1) | commit-engine | legacy `n_active==0→success` removed; n_peers≤1 short-circuit only (commit.c:330) | MET |
|
||||
| 5.1–5.3 config change commit barrier | barrier-rollout (config track) | `push_config_to_standbys` returns `-ETIMEDOUT`; caller forwards (config_repl.c:941,1115-1120); `committed_config_tag` gated (config_repl.c:963-969) | MET |
|
||||
| 6.1 replicate to all reachable standbys | (upstream ha-wal-replication, verified) | `repl_broadcast` full fan-out (unchanged) | MET |
|
||||
| 6.2 catch-up without per-step heartbeat | convergence-recovery | ack-driven continuation `ha_catchup_continuation_fn` config_repl.c:1732 (off-iomgr delaymgr) | MET |
|
||||
| 6.3, 6.4 snapshot fallback, re-include | (upstream, verified) | snapshot transfer + `repl_complete_catchup` (unchanged) | MET |
|
||||
| 6.5 lag never blocks majority confirmation | commit-engine | majority-median threshold only | MET |
|
||||
| 7.1 reject state changes without quorum | commit-engine + ha gate | dispatch gate + ACK-time enforcement (TOCTOU closed) | MET |
|
||||
| 7.2 log quorum-loss enter/exit | single-leader-role | `ha_log_quorum_transition` (ha.c:1085, 2076, 3969, 5054) | MET |
|
||||
| 7.3 serve status while degraded | (gate allowlist, verified) | dispatch-gate allowlist (status/replication) | MET |
|
||||
| 7.4 auto-resume on quorum restore | commit-engine + ha gate | stateless gate re-eval; restore edge logged ha.c:5054 | MET |
|
||||
| 7.5 standalone self-quorum | commit-engine | `n_peers <= 1` self-quorum commit.c:100,330; ha.c:2037 | MET |
|
||||
|
||||
**Umbrella criteria covered: 33/33 acceptance criteria MET in code.** One sub-task remains
|
||||
unticked within scope — `ha-convergence-recovery` task 8 (live 3-node E2E convergence test).
|
||||
This is test-evidence, not a missing invariant: the convergence MECHANISM is implemented and
|
||||
unit/integration-tested; the standing 3-node E2E is the same pending live test inherited from
|
||||
ha-wal-replication task 5.4. Tag: **MET (impl) / E2E-evidence pending.**
|
||||
|
||||
### Still-OPEN invariants (Deferred — owned by FUTURE specs, NOT umbrella scope)
|
||||
|
||||
These were declared out-of-scope by the umbrella (Boundary Context / Non-Goals) and remain
|
||||
unimplemented. They are **Deferred**, not regressions — the umbrella never promised them; it
|
||||
named the owning backstop.
|
||||
|
||||
| Open invariant | Why still open | Owning future spec (roadmap) | Effort | Risk |
|
||||
|---|---|---|---|---|
|
||||
| Worker-rejoin job-state resync (facts lost across disconnect/failover, never reached controller) | Primary-side RETAIN queue only covers facts the controller already received; worker notify is fire-and-forget with zero redelivery (bug-396). Designated backstop in umbrella Req 2.8 / boundary. | **worker-rejoin-resync** (roadmap, planned, no effort yet) | M | Medium — touches worker join/hello protocol + reconcile-through-barrier; shares the already-landed `mark_job_*` idempotency guards |
|
||||
| Automatic primary step-down on quorum loss | Umbrella satisfies CP by BLOCKING writes (Req 7.1); step-down is an availability/clarity optimization, explicitly out of scope. | **primary step-down-on-quorum-loss** (roadmap, optional CP follow-on) | S–M | Low — writes already blocked; step-down is additive role transition |
|
||||
| Read consistency / leader leases | Status queries may observe provisional/stale in-memory state (documented deviation); irrevocable-action egress already commit-gated (2.4). | **read-consistency / leader-leases** (roadmap, Raft §8) | M | Medium — read-only lease protocol; must not regress write path |
|
||||
| Entity-bearing snapshots ("Task 11" format) | Snapshot is a 16-byte LSN sentinel; demotion eviction uses self-restart instead. Not required by any umbrella criterion; strengthens §C option (iii) and faster recovery. | **entity-bearing-snapshots** (roadmap; dual-zarray COW already built/dormant) | M | Medium — serialization format + freeze/thaw wiring |
|
||||
| Job-array single-WAL-record | Each array element = one WAL/commit/push today; amplifies commit-engine load linearly. Correctness-neutral (each element still commits); performance only. | **job-array-single-wal-record** (roadmap) | M | Medium — atomic array commit + scheduler local expansion |
|
||||
|
||||
### Roll-up conclusion
|
||||
|
||||
The umbrella contract is **substantively SATISFIED**: all 33 acceptance criteria are MET in
|
||||
landed code across the four core sub-specs (commit-engine, barrier-rollout, single-leader-role,
|
||||
convergence-recovery) plus the durability sub-spec, which closed bug-661 (ACK≠durable) and the
|
||||
no-self-terminate promotion fence. The COMMITTING transitional state is fully removed; majority
|
||||
ACK (now durable-majority) is the hard commit barrier for all state-changing ops; facts are
|
||||
retained-and-retried; egress is commit-gated; demotion converges via truncate+self-restart.
|
||||
|
||||
Status: **PARTIAL** only in the umbrella-as-program sense — the four explicitly-deferred
|
||||
invariants above remain owned by named future specs, and one in-scope live 3-node E2E test
|
||||
(convergence-recovery task 8) is still pending. No in-scope invariant is unimplemented.
|
||||
|
||||
**Re-validation triggers still standing**: worker-rejoin-resync landing → re-verify idempotency
|
||||
guards + fact replay ordering; entity-bearing snapshots landing → re-evaluate demotion
|
||||
self-restart vs snapshot rebuild; any new state-changing opcode/WAL type → must be classified
|
||||
(the `wal_classify` debug abort at persist.c:2046 enforces this).
|
||||
|
|
|
|||
39
.kiro/specs/ha-elect-ctx-lifecycle/research.md
Normal file
39
.kiro/specs/ha-elect-ctx-lifecycle/research.md
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
# Research / Gap Analysis — ha-elect-ctx-lifecycle
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
> **Prerequisite not met:** this spec has only `brief.md` — **no `requirements.md`** (phase=`initialized`, `ready_for_implementation=false`). A full requirements↔codebase gap map cannot be produced. Run `/kiro-spec-requirements ha-elect-ctx-lifecycle` to formalize EARS requirements, then re-run `/kiro-validate-gap ha-elect-ctx-lifecycle`. The note below is a **pre-requirements gap sketch** derived from `brief.md` + a current code check.
|
||||
|
||||
### Current State (verified 2026-06-18)
|
||||
The deliberate-leak pattern the brief describes is **still present and unfixed** in `src/controller/ha.c`:
|
||||
- `ha.c:4229` — "Do NOT free a prior ctx (live vote endpoints may hold a bare pointer to it…)" immediately before `atomic_store(&ha->elect_ctx, mc)`. Each election/PreVote round `calloc`s a fresh `ha_elect_ctx_t` and intentionally leaks the previous one.
|
||||
- `ha.c:3978`, `ha.c:4830` — sibling "Do NOT free … may still reference / may already be running" comments on the async endpoint / delaymgr teardown paths (the same UAF caution).
|
||||
- No `ha_election_cleanup()` exists (only the aspirational comment the brief cites). Carrier read sites: `ha.c:123-124`, `234-235`, `903-904`.
|
||||
- Carrier captured by per-peer vote endpoints (`vote_ep_ctx_t.ctx = ectx`), freed asynchronously by `endpoint_fini` (FREEUP) on the iomgr thread — the root cause of the UAF hazard that makes a naïve free unsafe.
|
||||
|
||||
### Requirement-to-Asset Map (pre-requirements — tags: MET / Missing / Constraint)
|
||||
| Intended capability (from brief) | Current code | Tag |
|
||||
|----------------------------------|--------------|-----|
|
||||
| Carrier (`ha_elect_ctx_t`) freed when no endpoint references it | leaked by design at `ha.c:4229` | **Missing** |
|
||||
| Safe free vs async `endpoint_fini` (`vctx`) teardown | no refcount / no deferred-free ordering | **Missing** |
|
||||
| Applies to BOTH real-election and PreVote carrier paths (shared `ha->elect_ctx`) | both paths leak identically | **Missing / Constraint** (fix one without the other leaves the hazard) |
|
||||
| No regression of failover correctness (bug-339/bug-340) | current correctness intact; leak is memory-only | **Constraint** |
|
||||
|
||||
### Severity / Why deferred
|
||||
- **Pre-existing + bounded**: one small struct (`sizeof(ha_elect_ctx_t) + n_peers*votes[]`, hundreds of bytes for 3 nodes) per *failover/PreVote event* — a rare event, not steady-state. No unbounded growth under normal operation.
|
||||
- Surfaces as ~174 KB / ~1548 allocs under ASAN across the full controller suite's many simulated rounds; the test command uses `detect_leaks=0`, so it does not fail CI today.
|
||||
|
||||
### Implementation Approach Options (for the eventual design)
|
||||
- **Option A — Reference-count the carrier.** Increment per `vctx` that captures it, decrement on each `vctx` free, free at zero. Clean, localized; needs an atomic refcount on `ha_elect_ctx_t` and disciplined inc/dec at every capture/free site. Effort **M**, Risk **Medium** (must cover every `vctx` lifecycle exit, incl. error paths).
|
||||
- **Option B — Teardown-ordered deferred free.** Tie the free to confirmed vote-endpoint release (e.g. a generation/epoch the iomgr teardown signals back). Avoids per-capture bookkeeping but needs a reliable "all endpoints of round N released" signal. Effort **M–L**, Risk **Medium**.
|
||||
- **Option C — Hybrid.** Refcount as the primitive, plus an epoch guard so a superseded round's carrier is reclaimed promptly once its endpoints drain. Effort **L**, Risk **Medium**.
|
||||
|
||||
### Research-Needed (carry to design)
|
||||
- Enumerate **all** `vctx->ctx = ectx` capture sites and **all** `vctx` free sites (incl. `endpoint_fini` FREEUP, error/abort paths, `raft_cb_send_vote_request`, `ha_send_pre_vote_request`).
|
||||
- Confirm the memory-order discipline for the refcount vs the existing `atomic_store(&ha->elect_ctx, …, release)` / acquire reads.
|
||||
- Validation must run ASAN with **leak detection ENABLED** (success = zero carrier leaks across many rounds) — distinct from the current `detect_leaks=0` suite default.
|
||||
|
||||
### Effort / Risk (overall)
|
||||
- Effort **M** (single-area C change in `ha.c`/`ha.h`), Risk **Medium** (async-teardown UAF is the real hazard; must not regress bug-339/bug-340 failover correctness).
|
||||
|
||||
**Next step:** `/kiro-spec-requirements ha-elect-ctx-lifecycle` → `/kiro-spec-design` → implement under leak-detection-enabled ASAN.
|
||||
|
|
@ -307,3 +307,94 @@ ha-repl-hardening's backlog.
|
|||
- R2: Are sub-second `heartbeat_interval_ms` configurations a supported operating point? If yes, `liveness_poll_fn`/`primary_heartbeat_fn` must move from whole-second to ms arithmetic (`timespec` diff).
|
||||
- R3: Confirm with ops/runbook owners that DEBUG3-on-reset (vs Req 5.7's literal INFO) is the accepted trade-off; if so, amend Req 5.7 wording.
|
||||
- R4: Verify task 4.5 (`test-heartbeat-tolerance.sh`) has been executed against the post-e1517bb binary; the delaymgr ordering fix changed scheduling behavior after the last recorded run.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
Post-implementation audit. Tasks 15/15 `[x]`, `spec.json.phase = implemented`. This pass
|
||||
re-verifies every EARS acceptance criterion against the current on-disk code (the codebase
|
||||
evolved after the original 2026-06-10 audit — most notably the `ha-prevote` layer now wraps
|
||||
the failover trigger). All file:line refs verified via codegraph + grep on this date.
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
| AC | Status | Asset (file:line) | Notes |
|
||||
|----|--------|-------------------|-------|
|
||||
| 1.1 record last-activity timestamp | CLOSED | `ha_state_t.last_activity` (ha.c:1387 doc); written by `ha_on_heartbeat_received` ha.c:4886-4892 | CLOCK_MONOTONIC seconds |
|
||||
| 1.2 periodic elapsed check | CLOSED | `liveness_poll_fn` ha.c:4750-4801; re-arms at `heartbeat_interval_ms/2` (min 100ms) ha.c:4795-4798 | delaymgr task, no thread |
|
||||
| 1.3 any msg resets deadline | CLOSED | cluster.c dispatch: OPCode_nop:313-314, OPCode_hello/ctrlhello:278-279, wal-catchup:178-179, snapshot:210-211; replEntry path config_repl.c:703; config_repl.c:1452 | role-gated `CTRL_STANDBY`; bonus: also steps down from CTRL_ELECTING via `ha_discover_primary` |
|
||||
| 1.4 failover on timeout | CLOSED (semantics changed) | `liveness_poll_fn` ha.c:4784-4790 → `ha_on_primary_ctrl_fd_rdhup` ha.c:4639 | **No longer transitions directly to CTRL_ELECTING.** Now begins a PreVote round and stays CTRL_STANDBY until PreVote majority (ha.c:4655-4707, `ha_begin_failover`/`ha_start_prevote_async`). Behaviour is a superset of the EARS text — see Residual G1. |
|
||||
| 1.5 stop poll on role change | CLOSED | `ha_disarm_heartbeat_timeout_vtask` ha.c:4911-4919 (sets `cancelled`, nulls `s_liveness_ctx`); `liveness_poll_fn` self-skips when `role != CTRL_STANDBY` ha.c:4773-4774 | |
|
||||
| 1.6 no blocking thread | CLOSED | `hb_timeout_thread`/`hb_vtask_t`/`s_hb_vtask` removed (grep: 0 non-comment hits in src). Remaining `sem_timedwait` (ha.c:2380) is the **separate** `heartbeat_seen` semaphore for CTRL_INIT detection, explicitly retained (design.md:305-306) | |
|
||||
| 2.1 declare dead after `heartbeat_timeout_ms` | CLOSED | `liveness_poll_fn` ha.c:4782-4784 `(uint32_t)elapsed >= timeout_s` | second-granularity; see G2 |
|
||||
| 2.2 N=3 tolerate 2 misses | CLOSED (convention) | threshold uses existing config ratio | no new param (per Non-Goals) |
|
||||
| 2.3 N=2 tolerate 1 miss | CLOSED (convention) | same | |
|
||||
| 2.4 startup validation `timeout >= 2×interval` | CLOSED | `validate_heartbeat_config` startup_validation.c:143-167; wired pre-`ha_init` in cluster.c:1085-1095 (loud ERROR + stderr + `-EINVAL`) | |
|
||||
| 3.1 primary tracks idle per-standby | CLOSED | `standby_ep_ctx_t.last_push_time` ha.c:4939; read in `primary_heartbeat_fn` ha.c:5242 | |
|
||||
| 3.2 NOP after one idle interval | CLOSED | `primary_heartbeat_fn` ha.c:5242-5243 `elapsed >= interval_s` → `xce_nop_req` | |
|
||||
| 3.3 any send resets idle timer | PARTIAL | written on connect (`on_standby_connected` ha.c:5005-5010) and after NOP (ha.c:5252) | WAL/config pushes in repl.c do **not** write `last_push_time` (repl.c out-of-boundary). Effect: a busy primary still emits a redundant NOP up to once/interval. Benign (per spec.json note); see G3 |
|
||||
| 3.4 no redundant NOP within interval | PARTIAL | suppression honoured for NOP-vs-NOP ha.c:5243 | not suppressed by WAL/config traffic (same root as G3) |
|
||||
| 3.5 stop idle task on demotion | CLOSED | `primary_heartbeat_fn` role guard ha.c:5179-5182 returns 0 on non-PRIMARY | |
|
||||
| 4.1 ctrlhello counts as activity | CLOSED | cluster.c:271-282 (handle_ctrlhello → `ha_on_heartbeat_received`); config_repl.c:1452 | |
|
||||
| 4.2 committing window safe | CLOSED | ctrlhello resets `last_activity` before timeout; deadline armed from ctrlhello (config_repl.c:1450-1453). Additionally `ha_on_heartbeat_received` now ABANDONS an in-progress PreVote round on a fresh heartbeat (ha.c:4901-4908) — stronger race elimination than the original design | Note: `CTRL_PRIMARY_COMMITTING` state was removed project-wide (see memory: ha_consistency_invariants); AC 4.2's literal state name is stale but the guarantee holds |
|
||||
| 4.3 no premature failover within window | CLOSED | `liveness_poll_fn` fires only after full `heartbeat_timeout_ms` ha.c:4784 | |
|
||||
| 5.1 raft/vote/async-init unchanged | CLOSED (re-scoped) | raft.c untouched by this spec. NOTE: the failover *initiator* path changed under the separate `ha-prevote` spec (now PreVote-gated), not under this one | boundary held for THIS spec |
|
||||
| 5.2 EPOLLRDHUP immediate | CLOSED | `ha_on_primary_ctrl_fd_rdhup` ha.c:4639 still invoked directly from RDHUP epoll path; disarms poll first (ha.c:4645) | |
|
||||
| 5.3 both paths → same failover fn | CLOSED | RDHUP epoll and `liveness_poll_fn` both call `ha_on_primary_ctrl_fd_rdhup` ha.c:4788 | |
|
||||
| 5.4 RDHUP cancels pending poll | CLOSED | `ha_on_primary_ctrl_fd_rdhup` first step calls `ha_disarm_heartbeat_timeout_vtask` ha.c:4643-4645 | |
|
||||
| 5.5 single-node no interference | CLOSED | `primary_heartbeat_fn` only iterates `standby_endpoints` (empty for single-node) ha.c:5199 | |
|
||||
| 5.6 no poll in INIT/ELECTING | CLOSED | arm only from STANDBY callsites (config_repl.c:1447-1454 role-gated; `ha_prevote_rearm_liveness` ha.c:4068-4077); poll self-skips non-STANDBY ha.c:4773 | |
|
||||
| 5.7 INFO log arm/reset/fire | PARTIAL | arm: INFO ha.c:4859; fire: WARN ha.c:4786 (acceptable severity); reset: DEBUG3 ha.c:4892 (NOT INFO — deliberate anti-spam) | documented trade-off; see G4 |
|
||||
|
||||
Reqs met: **24/27 fully CLOSED**, 3 PARTIAL (3.3, 3.4, 5.7) — all pre-acknowledged benign
|
||||
in `spec.json.note` and the prior audit's R-list. No regressions found vs the 2026-06-10 pass.
|
||||
|
||||
### Residual / Deferred Gaps
|
||||
|
||||
- **G1 (design drift, not a defect) — RDHUP no longer goes straight to CTRL_ELECTING.**
|
||||
AC 1.4 / 1.5 / 5.x EARS text says the timeout "transitions to CTRL_ELECTING and initiates the
|
||||
RDHUP failover path." The implemented `ha_on_primary_ctrl_fd_rdhup` now begins a **PreVote
|
||||
round and stays CTRL_STANDBY** until a PreVote majority is won (ha.c:4655-4707), a change
|
||||
introduced by the downstream `ha-prevote` spec to prevent a transient blip from unseating a
|
||||
healthy primary. The liveness *detection* contract this spec owns is intact; only the
|
||||
*consequence* fans out through an extra safety gate. The spec docs were never updated to
|
||||
reflect this. Also: AC 4.2 references `CTRL_PRIMARY_COMMITTING`, a state since removed.
|
||||
|
||||
- **G2 (latent, sub-second configs) — whole-second timer arithmetic.** `liveness_poll_fn`
|
||||
(ha.c:4782 `timeout_s = ...ms/1000`) and `primary_heartbeat_fn` (ha.c:5195 `interval_s =
|
||||
...ms/1000`) truncate to whole seconds. With `heartbeat_interval_ms < 1000` the interval
|
||||
truncates to 0 (then floored: poll min 100ms, but NOP `interval_s` can be 0 → NOP every
|
||||
tick). `validate_heartbeat_config` enforces only the ratio, not a sub-second floor. Carried
|
||||
from prior audit R2 — unresolved.
|
||||
|
||||
- **G3 (Req 3.3/3.4 partial) — WAL/config pushes don't reset `last_push_time`.** `repl.c`
|
||||
send paths are out-of-boundary, so a busy primary emits one redundant NOP per interval per
|
||||
standby. Functionally harmless (NOP is cheap; standby treats it as activity). Pre-accepted.
|
||||
|
||||
- **G4 (Req 5.7 partial) — reset logged at DEBUG3 not INFO.** Deliberate to avoid per-heartbeat
|
||||
log spam. Needs a one-line wording amendment to AC 5.7 (or a runbook sign-off). Prior audit R3.
|
||||
|
||||
- **G5 (interaction w/ never-block-iomgr rule) — VERIFIED SAFE.** `liveness_poll_fn` and
|
||||
`primary_heartbeat_fn` run on the **delaymgr worker thread**, never in an iomgr vtask handler;
|
||||
the only blocking-capable call is `xce_nop_req` which is the non-blocking `xce_request` path
|
||||
(per CLAUDE.md HA invariants). `last_activity` is a plain `time_t` written from vtask context
|
||||
and read from delaymgr — atomic on LP64 (design.md:347). No iomgr-blocking risk introduced.
|
||||
|
||||
- **G6 (controller-never-self-terminate invariant) — VERIFIED COMPLIANT.** The timeout path
|
||||
fences/re-elects via PreVote and stays alive; the config-less (tag=0) guard (ha.c:4691-4700)
|
||||
parks the node as a passive voter in CTRL_STANDBY rather than exiting. No `exit/abort/FATAL`
|
||||
on the liveness path. Consistent with the repo's process-lifecycle design invariant.
|
||||
|
||||
### Options for the Open Items (G1, G2, G4)
|
||||
|
||||
| Item | Option A (recommended) | Option B | Option C | Effort / Risk |
|
||||
|------|------------------------|----------|----------|---------------|
|
||||
| G1 doc drift | Update requirements.md AC 1.4/4.2 + design.md to describe the PreVote-gated consequence and drop `CTRL_PRIMARY_COMMITTING`; add a cross-ref to the `ha-prevote` spec. Doc-only. | Re-open spec and add an EARS AC for the PreVote interaction. | Leave as-is; rely on `ha-prevote` spec to own the consequence. | A: ~30min / none (no code). B: ~2h / low. C: 0 / leaves spec text misleading for future readers. |
|
||||
| G2 sub-second | Add a sub-second floor check to `validate_heartbeat_config` (`interval_ms >= 1000`) — fail loud at startup. Tiny, matches existing validation pattern. | Convert both timers to `timespec` ms arithmetic to actually support sub-second. | Document "second-granularity only" in steering and close. | A: ~1h + 1 test / very low. B: ~half-day / medium (touches two hot timers; needs new tests). C: 0 / risk if an operator sets sub-second silently. |
|
||||
| G4 log level | Amend AC 5.7 wording to "DEBUG3 on reset" (one line). | Add an INFO log only on the *first* reset after an arm (state-change, not per-tick). | Keep DEBUG3, runbook sign-off. | A: ~5min / none. B: ~30min / none. C: 0 / minor. |
|
||||
|
||||
**Recommendation:** G1-A and G4-A are zero-risk doc fixes that close the audit cleanly.
|
||||
G2-A (startup floor) is the cheapest way to neutralize the only latent code hazard; defer
|
||||
G2-B unless sub-second heartbeats become a real operating requirement. No functional code
|
||||
change is required for correctness at the documented operating point (interval >= 1s).
|
||||
|
|
|
|||
|
|
@ -332,3 +332,73 @@ All seams appear to be in place. The tests should pass, but a build run (`/xce-b
|
|||
- **ASAN baseline diff**: run the controller suite at the pre-feature commit to confirm which of the 397 leaked allocations predate this spec (only needed if Option A is rejected).
|
||||
- **Watchdog ctx lifetime in production**: trace `heartbeat_timeout_ctx` ownership through cancel/re-arm cycles on a long-lived standby (ha.c:3337–3449).
|
||||
- **Uncommitted raft.c changes**: determine which spec owns the `ha_init_raft` extraction and the `primary_ctrl_fd >= 0` vote-gate tightening (interacts with Req 3.1's designation semantics), and validate against the TLA+ checklist in `src/controller/CLAUDE.md`.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
> POST-IMPLEMENTATION GAP AUDIT. Scope: requirements.md (Reqs 1–5, 19 acceptance criteria) vs on-disk code on `persistent_ha`. tasks.md = 17/17 `[x]`; spec.json `phase: "implemented"`. Verified via codegraph_explore over the live source. This pass re-confirms the 2026-06-09 findings against code that has since absorbed adjacent specs (ha-prevote, ha-repl-hardening, ha-commit-durability, ha-convergence-recovery).
|
||||
|
||||
### 1. What changed since the last audit (2026-06-09)
|
||||
|
||||
- **Req 5.5 ASAN leak — CLOSED.** spec.json note (2026-06-10): test-fixture leaks fixed via `free(ep0)/free(ep1)` in the two affected tests; controller suite green under ASAN. The prior "Partial" verdict is upgraded to **Implemented**.
|
||||
- **spec.json phase drift — FIXED.** Now `"implemented"` (was `"tasks-generated"`).
|
||||
- **Dispatcher evolved (cross-spec, still compliant).** `peer_ctrl_msg_handler` (ha.c:3600) now routes `CTRL_PRIMARY` → `repl_ack_decode_and_dispatch` (ha.c:3618) and added a `CTRL_STANDBY && ha->raft.pre_vote_in_progress` → `pre_vote_reply_decode_and_dispatch` branch (ha.c:3610). The original design's `CTRL_PRIMARY_COMMITTING` case is gone because that role was removed by ha-consistency-invariants. Still "set once, never reassigned" — Req 1.2 intact.
|
||||
- **`handle_ctrlhello` strengthened.** Now lives in config_repl.c:1436 (design.md said cluster.c — superseded, tasks.md was right). On first ctrlhello in `CTRL_STANDBY` with `primary_ctrl_fd == -1` it sets `primary_ctrl_fd = ep->fd`, calls `ha_on_heartbeat_received`, AND arms `ha_arm_heartbeat_timeout_vtask` (ha.c:1447–1455). Exceeds original Req 3.1 (which only required designation) — also covers 3.3 from the receive path.
|
||||
- **Compliant superset on the primary side.** Late/failed peers are reconnected by `primary_heartbeat_fn` → `ha_connect_to_peer` (ha.c:5103) via fresh outbound connects with `standby_ep_err_handler_primary`. This is the primary acting as primary — does NOT violate Req 2.1 (standby passivity) and respects Req 4.2's "in that election round" scope.
|
||||
|
||||
### 2. Requirement-to-Asset Map
|
||||
|
||||
| Req | Summary | Code Asset (file:line) | Tag |
|
||||
|-----|---------|------------------------|-----|
|
||||
| 1.1 | One outbound TLS connection per peer for vote RPC | `raft_cb_send_vote_request` ha.c (+ PreVote `raft_cb_send_pre_vote_request` ha.c:223); SSL via `ha->client_ssl_ctx` | CLOSED |
|
||||
| 1.2 | Reuse surviving vote conn as persistent peer channel | `peer_ctrl_msg_handler` ha.c:3600 (set once `on_vote_connected`, never reassigned) | CLOSED |
|
||||
| 1.3 | Winner promotes vote conns → standby channels | `raft_cb_become_leader` promotion loop ha.c:~1056 (`zarray_store(standby_endpoints, votes[i].ep)`) | CLOSED |
|
||||
| 1.4 | Loser closes all outbound vote conns | `raft_cb_become_follower` close loop ha.c:~1179 (`endpoint_fini(votes[i].ep)`; NULL-out) | CLOSED |
|
||||
| 2.1 | Standby initiates no outbound TCP to peers | `fetch_config_from_primary` + `ha_find_and_connect_to_primary` deleted; passive contract myapp.c:312 | CLOSED |
|
||||
| 2.2 | Standby processes ctrlhello/config/WAL on inbound conn | cluster.c `myreq_handler` → `handle_ctrlhello` config_repl.c:1436; config/WAL handlers unchanged | CLOSED |
|
||||
| 2.3 | No inbound comm within timeout → new election | `ha_arm_heartbeat_timeout_vtask` armed in `raft_cb_become_follower` (~ha.c:1238) + `handle_ctrlhello` (config_repl.c:1453) | CLOSED |
|
||||
| 2.4 | No tier-1/2/3 discovery on standby | `ha_find_and_connect_to_primary` removed entirely (exceeds design "remain for future use") | CLOSED |
|
||||
| 3.1 | First ctrlhello → designate primary control channel | `handle_ctrlhello` config_repl.c:1447–1455, idempotency guard `primary_ctrl_fd == -1` | CLOSED |
|
||||
| 3.2 | RDHUP detected in one epoll cycle → CTRL_ELECTING | iomgr `io_handler`/`handle_error` netio.c:444–525 → err-handler → re-election | CLOSED |
|
||||
| 3.3 | Watchdog same timeout semantics as `primary_ctrl_fd` | Same `ha_arm_heartbeat_timeout_vtask` (`heartbeat_timeout_ms`) ha.c:4825 | CLOSED |
|
||||
| 4.1 | Explicit close of vote endpoints on loss | `endpoint_fini` + NULL-out in `raft_cb_become_follower` (~ha.c:1181) | CLOSED |
|
||||
| 4.2 | Failed vote conn → denied vote, no reconnect that round | `vote_ep_err_handler` ha.c:3937: ELECTING→`handle_request_vote_reply(grant=0)`; PRIMARY→remove from standby_endpoints + `repl_remove_standby` | CLOSED |
|
||||
| 4.3 | New election → fresh outbound connections | fresh `ha_elect_ctx_t`/eps per round in send-vote callback | CLOSED |
|
||||
| 5.1 | Config push via promoted connections | `push_config_transport_only` config_repl.c:879 over populated `standby_endpoints` | CLOSED |
|
||||
| 5.2 | WAL replication via promoted connections | `repl_ack_decode_and_dispatch` → `ha_repl_ack_received`; promoted ep registered via `repl_add`/`repl_remove_standby` | CLOSED |
|
||||
| 5.3 | Single-node self-elect, no peer connections | raft single-node shortcircuit (unchanged); `ectx`/NULL guards in both callbacks | CLOSED |
|
||||
| 5.4 | Committing timeout → degraded PRIMARY | promotion-fence/own-term-gate machinery (`ha_arm_own_term_gate`, fence timer ha.c:692) — superset of original `committing_timeout_fn` | CLOSED |
|
||||
| 5.5 | No new ASAN errors vs baseline | Test-fixture leaks fixed 2026-06-10 (spec.json note); suite green under ASAN | CLOSED |
|
||||
|
||||
**Score: 19 of 19 acceptance criteria CLOSED.** (Up from 18/19 + 1 Partial on 2026-06-09.)
|
||||
|
||||
### 3. Residual gaps
|
||||
|
||||
No functional or correctness gaps remain. Only documentation/metadata drift (cosmetic, non-blocking):
|
||||
|
||||
- **R1 (LOW, doc):** Stale comment ha.c:~4498 "Transition to CTRL_STANDBY so myapp_run schedules fetch_config_from_primary" — the function no longer exists; the standby path is passive. Comment is now misleading.
|
||||
- **R2 (LOW, doc):** `raft_cb_become_follower` watchdog comment still says "election_timeout_ms" while the armed vtask uses `heartbeat_timeout_ms` (`ha_arm_heartbeat_timeout_vtask`). Same stale wording flagged on 2026-06-09; still present.
|
||||
- **R3 (LOW, doc):** Comment refs to `fetch_config_from_primary()` / `ha_find_and_connect_to_primary` persist in main.c:218, cluster.c:1209, ha.h:1091 as historical breadcrumbs. Harmless but reference removed symbols.
|
||||
- **R4 (LOW, design drift):** design.md still (a) places `handle_ctrlhello` in cluster.c (actual: config_repl.c) and (b) says `ha_find_and_connect_to_primary` "remains for future use" (actual: removed). design.md should be annotated "superseded" — tasks.md/spec.json are correct.
|
||||
|
||||
### 4. Integration risks (forward-looking, not gaps)
|
||||
|
||||
- **Shared-asset coupling:** `peer_ctrl_msg_handler`, `standby_endpoints`, and the promoted-connection model are now load-bearing for ha-prevote (PreVote reply routing), ha-repl-hardening (heartbeat decision loop over promoted eps), and ha-commit-durability (promotion fence + own-term gate run on the primary that owns these eps). Any future change to the dispatcher's role switch or to `standby_endpoints` population must re-run those specs' suites. The design.md "Revalidation Triggers" list already names these — keep it authoritative.
|
||||
- **vctx lifetime:** `vote_ep_err_handler` deliberately does NOT free `vctx` (ha.c:3978) — it is freed in `ha_election_cleanup`. With promoted connections outliving the election, confirm `ha_election_cleanup` (or endpoint teardown) still reaches every promoted ep's vctx on a long-lived primary. Currently consistent, but it is the one lifetime invariant most exposed to drift.
|
||||
|
||||
### 5. Options for the open (cosmetic-only) work
|
||||
|
||||
**Option A — Comment/design hygiene only (RECOMMENDED).** Fix R1–R4 in place (4 comment edits + 2 design.md annotations). Effort: S (~30 min). Risk: Low (no code-behavior change). Closes the spec cleanly with zero outstanding items.
|
||||
|
||||
**Option B — Leave as-is, note in steering.** Record the stale-comment debt in `.wolf/memory.md` / a docs backlog and move on, since all 19 criteria are functionally CLOSED. Effort: XS. Risk: Low, but the misleading ha.c:4498 comment can mislead a future reader on the standby path.
|
||||
|
||||
**Option C — Bundle into a cross-spec doc-sweep.** Defer R1–R4 to a single documentation pass across the HA spec family (several specs carry similar removed-symbol breadcrumbs). Effort: M (coordinated). Risk: Low; best ROI if other specs share the debt.
|
||||
|
||||
| Item | Effort | Risk |
|
||||
|------|--------|------|
|
||||
| R1–R3 stale comment fixes | S | Low |
|
||||
| R4 design.md "superseded" annotations | S | Low |
|
||||
| Optional: vctx-lifetime trace on long-lived primary (integration risk #2) | S | Low |
|
||||
|
||||
**Verdict: feature COMPLETE — 19/19 acceptance criteria CLOSED. Remaining work is documentation hygiene only (Option A recommended).**
|
||||
|
|
|
|||
|
|
@ -251,3 +251,116 @@ four `test_invariant_*` tests are the mechanical evidence for premises (1)–(4)
|
|||
and the existing failover/rejoin regression suites remain green with PreVote
|
||||
enabled (controller suite 563 pass / 0 fail after task 4.2). Requirements
|
||||
covered: 5.2, 5.3, 6.1, 6.2, 6.3, 6.4.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
**Method.** Verified every EARS acceptance criterion in `requirements.md` against
|
||||
the current on-disk source via codegraph + grep. tasks.md shows phase
|
||||
`tasks-generated` with 6 unchecked `[ ]` and 12 checked `[x]`; this audit
|
||||
classifies each `[ ]` and tags each requirement MET/Missing/Unknown/Constraint.
|
||||
|
||||
### STATUS: CLOSED — requirements met 7/7, all 12 leaf tasks implemented + tested
|
||||
|
||||
Every production seam the design names exists, is wired into the real failover
|
||||
path, and is exercised by tests. The 6 unchecked boxes are **group-header
|
||||
artifacts**, not work.
|
||||
|
||||
### Open-task classification (the 6 unchecked `[ ]`)
|
||||
|
||||
All 6 unchecked boxes are the **phase/group-header lines** (task numbers `1`–`6`,
|
||||
the section titles). Every one of their leaf sub-tasks is checked `[x]`:
|
||||
|
||||
| `[ ]` header | Children (all `[x]`) | Classification |
|
||||
|---|---|---|
|
||||
| 1. Foundation: wire schema + raft scaffolding | 1.1, 1.2 | group-header-artifact |
|
||||
| 2. Core: raft-library pure decision + tally | 2.1, 2.2, 2.3 | group-header-artifact |
|
||||
| 3. Core: controller voter-side dispatch | 3.1 | group-header-artifact |
|
||||
| 4. Core: candidate orchestration + trigger restructure | 4.1, 4.2 | group-header-artifact |
|
||||
| 5. Integration: transport wiring + E2E | 5.1, 5.2, 5.3 | group-header-artifact |
|
||||
| 6. Validation: safety invariants + regression guard | 6.1 | group-header-artifact |
|
||||
|
||||
**Genuine-gap: 0. Deferred: 0. Artifact: 6.** A group header is complete by
|
||||
definition once all its children are done; these were never intended as
|
||||
separately-checkable work items. No effort/risk applies (nothing to build).
|
||||
|
||||
### Requirement-to-Asset map
|
||||
|
||||
| Req | Criterion (summary) | Asset (file:line) | Tag |
|
||||
|---|---|---|---|
|
||||
| 1.1 | PreVote before term increment on trigger | `ha_begin_failover` ha.c:4606; `ha_start_prevote_async` ha.c:4193; triggers `ha_on_primary_ctrl_fd_rdhup` ha.c:4639 / `liveness_poll_fn` (calls rdhup) ha.c:~4789 | MET |
|
||||
| 1.2 | No term/vote mutation during round | `raft_begin_pre_vote` raft.c:232 (writes only `pre_vote_*` tally; no term/voted_for/role/persist) | MET |
|
||||
| 1.3 | Request prospective term+1, not persisted | `raft_begin_pre_vote` raft.c:243 (`prospective_term = s->term + 1`, local only) | MET |
|
||||
| 1.4 | No majority → no real election | `pre_vote_reply_dispatch` ha.c:4296 LOST branch (stay STANDBY, re-arm) | MET |
|
||||
| 1.5 | Majority → start real election | `pre_vote_reply_dispatch` WON → `ha_enter_real_election` ha.c:4103 → unchanged `ha_start_election_async`/`raft_start_election` | MET |
|
||||
| 2.1 | Grant only if real-vote freshness gate passes | cluster.c:480 `ha_vote_self_freshness` floor + cluster.c:512 `raft_on_pre_vote_request` | MET |
|
||||
| 2.2 | Fail freshness → deny | cluster.c:482-506 `freshness_ok` → `deny_reason="freshness"` | MET |
|
||||
| 2.3 | Grant/deny records no vote, no term | `raft_on_pre_vote_request` raft.c:193 (pure); reply carries unchanged term cluster.c:500/556 | MET |
|
||||
| 2.4 | PreVote does not consume the real vote | `raft_on_pre_vote_request` writes no `voted_for`; asserted `test_genuine_failover...` test_ha_prevote.c:709 | MET |
|
||||
| 3.1 | Deny while leader believed alive | `raft_cb_vote_eligible` ha.c:299 (STANDBY + fresh heartbeat + primary_ctrl_fd>=0 → deny), reused by PreVote | MET |
|
||||
| 3.2 | Past self-election threshold → freshness-only eval | `raft_cb_vote_eligible` falls through to grant once heartbeat stale; `test_genuine_failover_stale_peer_grants_and_elects` | MET |
|
||||
| 3.3 | Primary denies PreVotes | `raft_cb_vote_eligible` ha.c:307 (CTRL_PRIMARY → 0) | MET |
|
||||
| 4.1 | Frozen-resumed gets no majority | scenario test `test_ha_prevote.c` (Angle B deny) + cluster.c deny path | MET |
|
||||
| 4.2 | Re-attach on next heartbeat, no term change | `ha_on_heartbeat_received` ha.c:4886-4909 (cancel round mid-flight, stay STANDBY); `test_heartbeat_mid_round_cancels_prevote` test_ha_failover_prevote.c:198 | MET |
|
||||
| 4.3 | Healthy primary not demoted/restarted | trigger defers truncate/CTRL_ELECTING to WON only (ha.c:4655-4662) | MET |
|
||||
| 4.4 | Term unchanged, no re-election cycle | candidate stays CTRL_STANDBY through round; round state in `raft_state_t` only | MET |
|
||||
| 5.1 | True death → PreVotes granted → elect | `test_genuine_failover_stale_peer_grants_and_elects` test_ha_prevote.c:680 | MET |
|
||||
| 5.2 | Partition heal → adopt higher term | `raft_on_vote_reply` raft.c:346 (unchanged real-vote path) | Constraint (unchanged, preserved) |
|
||||
| 5.3 | No regression of bug-339/bug-340 | real-vote path byte-for-byte unchanged; `test_invariant_real_vote_higher_term_adopt_unchanged` (RN-1) | MET |
|
||||
| 5.4 | Complete within election-timeout bounds | round timeout `election_timeout_ms` ha.c:4258; rdhup disarms liveness first | MET |
|
||||
| 6.1 | Election Safety preserved | PreVote fires no `on_become_leader`; RN-1 + `test_invariant_prevote_won_grants_no_leadership` | MET |
|
||||
| 6.2 | Term monotonicity; PreVote never changes term | RN-1 + `test_invariant_prevote_sequence_pure_no_persist` | MET |
|
||||
| 6.3 | Leader Completeness, monotonic commit index | freshness floor reused; commit path untouched | Constraint (unchanged, preserved) |
|
||||
| 6.4 | Consistent with raft.tla + checklist | RN-1 argument (research.md §RN-1) + invariant tests | MET |
|
||||
| 7.1 | PreVote over existing transport, no real-vote disruption | new opcodes `preVoteRequest @38`/`preVoteReply @39` controller.capnp:49-50, reuse vote structs; reply routed via `peer_ctrl_msg_handler` ha.c:3610 (pre_vote_in_progress gate) → `pre_vote_reply_decode_and_dispatch` ha.c:3340 | MET |
|
||||
| 7.2 | Log denials w/ candidate + reason | cluster.c:546-552 `INFO("HA: PreVote DENIED candidate=... reason=...")` (gate-order-faithful reason) | MET |
|
||||
| 7.3 | Log abandoned campaign distinctly | `pre_vote_reply_dispatch` LOST branch ha.c:4313 INFO; round-timeout fn | MET |
|
||||
|
||||
**Tally: MET 25, Constraint (preserved-unchanged) 2 (5.2, 6.3), Missing 0, Unknown 0.**
|
||||
|
||||
### Notable implementation facts (beyond the design baseline)
|
||||
|
||||
- **Reply routing** is NOT a dedicated `OPCode_preVoteReply` switch case in
|
||||
cluster.c (as task 5.1 literally phrased it). Instead `peer_ctrl_msg_handler`
|
||||
(ha.c:3600) routes by role+state: `CTRL_STANDBY && pre_vote_in_progress` →
|
||||
`pre_vote_reply_decode_and_dispatch` → `pre_vote_reply_dispatch`. The request
|
||||
side does emit/reply over the dedicated `OPCode_preVoteReply` opcode
|
||||
(cluster.c:564). Functionally equivalent to the design intent; the dispatch
|
||||
discriminator is round-state rather than a third reply opcode case.
|
||||
- **Two extra robustness mechanisms** were added beyond the spec, both benign
|
||||
and in-scope-adjacent: (a) a **per-node stagger** (`ha_election_stagger_ms`
|
||||
→ `ha_failover_stagger_fn`, ha.c:4586) before starting PreVote to avoid
|
||||
lock-step split votes (bug-606); (b) a retained **config-less-node guard**
|
||||
(tag==0 && n_peers>1 → voter-only, ha.c:4693) that short-circuits before the
|
||||
round (ha-config-bootstrap Req 2.8).
|
||||
- **Test coverage** spans three files: `src/lib/testsuite/test_raft.c` (pure
|
||||
decision/tally + 4 invariant tests), `src/controller/testsuite/test_ha_prevote.c`
|
||||
(voter decide + frozen-rejoin + genuine-failover scenarios),
|
||||
`src/controller/testsuite/test_ha_failover_prevote.c` (trigger rewire:
|
||||
rdhup/liveness start PreVote not election, heartbeat-mid-round cancel,
|
||||
WON→real-election, config-less-node no-PreVote). Per Implementation Notes
|
||||
(task 4.2): controller suite 563 PASS / 0 FAIL with PreVote live.
|
||||
|
||||
### Carry-forward (NOT a ha-prevote gap)
|
||||
|
||||
The deliberate per-round `elect_ctx` ASAN leak (`ha_start_prevote_async`
|
||||
allocates a metadata `ha_elect_ctx_t` that is intentionally not freed because
|
||||
live vote endpoints may hold a bare pointer — same UAF caution as the rdhup
|
||||
path) is pre-existing, bounded (~one per round), and explicitly carried into a
|
||||
separate follow-up spec `.kiro/specs/ha-elect-ctx-lifecycle/`. It is out of
|
||||
scope for ha-prevote and does not affect any ha-prevote requirement.
|
||||
|
||||
### Options for the (zero) genuine remaining work
|
||||
|
||||
- **Option A (recommended):** Mark ha-prevote CLOSED. Optionally check the 6
|
||||
group-header boxes for bookkeeping cleanliness (cosmetic; all children done).
|
||||
**Effort: S. Risk: none.**
|
||||
- **Option B:** Run `/kiro-verify-completion ha-prevote` to mechanically confirm
|
||||
every acceptance criterion against the suite before flipping spec.json phase
|
||||
to a completed state. **Effort: S. Risk: low** (known `test_commit_barrier_reply`
|
||||
300s TIMEOUT in the full controller suite — use the targeted run and check for
|
||||
zero `:FAIL`, per Implementation Note 3.1).
|
||||
- **Option C:** No action; treat the unchecked headers as a tasks.md rendering
|
||||
convention. **Effort: none. Risk: none** (the only downside is `[ ]` headers
|
||||
reading as "incomplete" to a casual `/kiro-spec-status` glance).
|
||||
|
|
|
|||
|
|
@ -324,3 +324,143 @@ flag.
|
|||
serialization, and can `send_wal_batch` be reused for it.
|
||||
- Req 5.5 "durably written" on the standby: is in-memory entity restore + in-memory applied_lsn
|
||||
acceptable (current documented design), or should batch apply fsync a standby-side WAL copy.
|
||||
|
||||
---
|
||||
|
||||
# Gap Re-Audit (2026-06-18)
|
||||
|
||||
> Post-implementation gap audit. spec.json `phase=implemented`; all 41 sub-tasks `[x]`
|
||||
> (1.1–7.6 incl. the reopened payload data-plane series). This pass re-verifies every EARS
|
||||
> criterion against the code **as on disk** — specifically whether the two CRITICAL data-plane
|
||||
> gaps and the small correctness gaps from the 2026-06-09 pass are now closed, and whether the
|
||||
> COMMITTING-gate requirements still describe the live code after `ha-single-leader-role` landed.
|
||||
|
||||
## Headline Finding
|
||||
|
||||
**Both 2026-06-09 CRITICAL blockers are CLOSED. The data plane is real.**
|
||||
|
||||
- WAL catch-up now applies payload-bearing entries. `ha_apply_single_entry` (ha.c:1558) does
|
||||
persist-before-apply: `persist_wal_write_replicated(p, e->lsn, type, e->raw_payload, len)`
|
||||
(fsync, no re-broadcast, idempotent) THEN `apply_wal_entry(p, &hdr, e->raw_payload)`.
|
||||
`apply_wal_entry` (persist.c:1305) restores entities by type — `WAL_JOB_SUBMIT`→`restore_job`,
|
||||
job-state→`restore_job_state`, `WAL_WORKER_CTRL`→`worker_from_capnp`,
|
||||
`WAL_QUEUE_CTRL`→`restore_queue_state` (persist.c:1342-1389). Header-only application is gone.
|
||||
- Snapshot payload bytes are transmitted. `ha_send_snapshot_to_standby` (config_repl.c:1857)
|
||||
stats the real file, chunks 64KB, and ships each chunk via `xce_snapshot_chunk_req_raw(ep,
|
||||
&chunk, buf, chunk_len)` (config_repl.c:1928) — raw `Data` bytes, not NULL. Standby
|
||||
`ha_handle_snapshot_chunk` (ha.c:1683→ via `handle_snapshot_transfer` cluster.c:196, dispatched
|
||||
at cluster.c:589) writes chunks to `snap.part`, verifies total size on `is_last`, atomic-renames
|
||||
to `snap-{lsn}.bin`, loads via `persist_load_snapshot`, sets `applied_lsn`.
|
||||
|
||||
**The Majority Consistency Invariant is now actually established, not merely tracked.** A standby
|
||||
that reports converged holds the entity state behind its `applied_lsn`.
|
||||
|
||||
## Architectural supersession (read before re-opening any 6.x task)
|
||||
|
||||
`CTRL_PRIMARY_COMMITTING` **no longer exists** — `grep -rn CTRL_PRIMARY_COMMITTING src/controller`
|
||||
returns zero hits. `ha-single-leader-role` removed the transitional COMMITTING role:
|
||||
`raft_cb_become_leader` now stores `CTRL_PRIMARY` directly (ha.c:1034) with the comment "there is
|
||||
no transitional COMMITTING state." New-leader safety is provided by a different, stronger
|
||||
mechanism than this spec's heartbeat-convergence gate:
|
||||
|
||||
1. **Own-term WAL_NOOP + writable_lsn gate** (`ha_arm_own_term_gate` ha.c:2259) — every WAIT
|
||||
completion is withheld until `committed_lsn >= own-term noop LSN`, i.e. until this term has
|
||||
committed an entry on a majority.
|
||||
2. **Per-operation write quorum** (`ha_dispatch_gate` ha.c:2061 → `ha_has_write_quorum` ha.c:2031)
|
||||
— state-changing opcodes return `-XCE_ERR_NOT_READY` unless `repl_active_standby_count >=
|
||||
n_peers/2`; replication and read-only/fact-bearing opcodes are always admitted (allowlist at
|
||||
ha.c:2078-2106).
|
||||
3. **Vote-derived safe commit point** (`ha_compute_safe_committed_lsn` ha.c:480, called from
|
||||
raft_cb_become_leader ha.c:908) — `safe_committed_lsn` = majority-threshold element of granted
|
||||
voters' LSNs; `persist_truncate_wal_above(safe_lsn)` + `committed_lsn=safe_lsn` (ha.c:914).
|
||||
|
||||
Net effect: the *intent* of Req 6 (don't serve un-replicated state; gate on majority) is met
|
||||
**more strongly and continuously** than the original timeout-bounded COMMITTING design — it is now
|
||||
enforced per-operation for the life of the term, not just at startup. The original Req 2.3 / 6.1–6.4
|
||||
wording (COMMITTING role, 60s `committing_timeout_ms` degraded fallback, immediate
|
||||
COMMITTING→PRIMARY on majority) describes a design that was deliberately replaced. This is a
|
||||
**documentation drift**, not a functional gap.
|
||||
|
||||
## Requirement-to-Asset Map (re-verified)
|
||||
|
||||
| Req / AC | Asset (file:line) | Status |
|
||||
|---|---|---|
|
||||
| **1.1** heartbeat reply carries (tag,lsn) | `OPCode_nop` handler cluster.c:300-301; populates from `myconf->config_version_tag` + `applied_lsn` | CLOSED |
|
||||
| **1.2** primary records + evaluates | `repl_ack_decode_and_dispatch` → `ha_process_standby_state` ha.c:1851 | CLOSED |
|
||||
| **1.3** convergence criteria tag≥ AND lsn≥ | ha.c:1864-1865 | CLOSED |
|
||||
| **1.4** tag=0 before config push | reads live `config_version_tag` (0 pre-push) | CLOSED |
|
||||
| **2.1** init tracking on term entry | **FIXED** — `raft_cb_become_leader` memsets `standby_reported_tag/_lsn/_converged` (ha.c:1043-1045) on CTRL_PRIMARY entry. Closes 2026-06-09 Gap 3.1. | CLOSED |
|
||||
| **2.2** update + re-evaluate on reply | `ha_process_standby_state` ha.c:1851-1876 | CLOSED |
|
||||
| **2.3** majority threshold | **RECONCILED** to `n_peers/2` (C int div) per task 7.5 + Decision Log 2026-06-09. `ha_has_write_quorum` uses `connected >= n_peers/2` (ha.c:2058); `safe_committed_lsn` uses `n_granted/2` (ha.c:527). Three-doc disagreement resolved. | CLOSED |
|
||||
| **2.4** INFO on promotion/converged | ha.c:1869-1875 (per-standby converged INFO) | CLOSED |
|
||||
| **2.5** regression → not-converged | ha.c:1863-1866 (recomputed every reply) | CLOSED |
|
||||
| **3.1** decode + apply WAL entry **with payload** | **FIXED** — `ha_apply_single_entry` ha.c:1558-1589 passes `raw_payload` to `apply_wal_entry`; entities restored persist.c:1342-1389. Closes 2026-06-09 CRITICAL. | CLOSED |
|
||||
| **3.2** advance applied_lsn | `ha_advance_applied_lsn` ha.c:1541 (atomic monotonic); batch advances after full batch (ha.c:1656-1659) | CLOSED |
|
||||
| **3.3** ERROR + skip on failure | ha.c:1633-1647, 1666-1672 (NULL/failed sub-entries skipped, no crash) | CLOSED |
|
||||
| **3.4** stays STANDBY, no job ops | `ha_dispatch_gate` STANDBY case | CLOSED |
|
||||
| **3.5** LSN order | sequential batch iteration + persist-before-apply | CLOSED |
|
||||
| **4.1** fresh checkpoint + transfer | **FIXED** — `ha_dispatch_repl_action_for_standby` calls `persist_checkpoint(&c->persist)` before send (ha.c:1961), -EBUSY falls back to existing. Closes 2026-06-09 Gap 3.3. | CLOSED |
|
||||
| **4.2** load snapshot, set applied_lsn | `ha_handle_snapshot_chunk` ha.c:1683 (verify→rename→`persist_load_snapshot`→applied_lsn); reachable in production (real payload) | CLOSED |
|
||||
| **4.3** no job ops until loaded | dispatch gate (standby role) | CLOSED |
|
||||
| **4.4** retry via reconnect / discard stale | **FIXED (cleaner than proposed)** — `chunk_index==0` unlinks any stale `snap.part` (ha.c:1693, 1708-1713) instead of relying on a reconnect-time unlink. Verified by `test_snapshot_recv_reconnect_discards_snappart` (test_config_repl.c:2045). Closes 2026-06-09 Gap 3.2. | CLOSED |
|
||||
| **4.5** single-node: no snapshot | no standbys → decision loop never fires | CLOSED |
|
||||
| **5.1** multiple entries/message | `entries :List(WalCatchupEntry)` + `send_wal_batch` (config_repl.c) | CLOSED |
|
||||
| **5.2** configurable max batch | `WAL_STREAM_BATCH_ENTRIES`=256 / `_BYTES`=64KB compile-time #defines | CLOSED (minor: limits are compile-time, not runtime ha_config_t — Req says "configurable maximum"; defensible) |
|
||||
| **5.3** apply in order, advance after batch | `ha_apply_wal_catchup_entry` ha.c:1615-1681 (`last_good_lsn` post-batch advance) — now with real payload | CLOSED |
|
||||
| **5.4** partial batch sent immediately | config_repl.c batch flush on range end | CLOSED |
|
||||
| **5.5** ordering + durability | Ordering preserved; durability now STRONGER than original design — `ha_apply_single_entry` fsyncs via `persist_wal_write_replicated` BEFORE apply/ACK (ha.c:1570). Supersedes the "in-memory only" caveat for catch-up. | CLOSED (exceeds spec) |
|
||||
| **6.1** heartbeats continue / gate on majority | superseded: continuous per-op `ha_has_write_quorum` (ha.c:2031) replaces COMMITTING-only check | CLOSED (via supersession) |
|
||||
| **6.2** NOT_READY for non-repl opcodes | `ha_dispatch_gate` ha.c:2061-2110 (allowlist + `-XCE_ERR_NOT_READY` default) — now enforced for the whole term, not just COMMITTING | CLOSED (stronger) |
|
||||
| **6.3** immediate promotion on majority | superseded by own-term gate + safe_committed_lsn (no COMMITTING dwell to exit) | CLOSED (via supersession) |
|
||||
| **6.4** timeout → WARN + degraded | **OBSOLETE** — no COMMITTING state ⇒ no `committing_timeout_ms`. Degraded operation handled by write-quorum gate (rejects writes, keeps serving reads) rather than a one-shot 60s promotion fallback. | N/A (superseded) |
|
||||
| **6.5** single-node immediate | `ha_has_write_quorum` returns true when `n_peers<=1` (ha.c:2037); safe_committed_lsn single-node path ha.c:486 | CLOSED |
|
||||
| **7.1** catch-up start INFO | config_repl.c (send_wal_batch start log) | CLOSED |
|
||||
| **7.2** converged INFO | ha.c:1869-1875 | CLOSED |
|
||||
| **7.3** snapshot start/complete INFO | config_repl.c:1874-1876, 1939-1941 | CLOSED |
|
||||
| **7.4** majority INFO | per-standby converged INFO (ha.c:1870); promotion is direct-to-PRIMARY INFO (ha.c:1036) | CLOSED |
|
||||
| **7.5** timeout WARN | OBSOLETE with COMMITTING removal; quorum-loss edge logged once via `ha_log_quorum_transition` (ha.c:2076) instead | N/A (superseded by quorum-transition logging) |
|
||||
| **Flow 0** safe commit point | `ha_compute_safe_committed_lsn` ha.c:480-540; truncate+baseline raft_cb_become_leader ha.c:898-923 | CLOSED |
|
||||
|
||||
**Score: 31/33 original acceptance criteria CLOSED; 2 (6.4, 7.5) OBSOLETE by deliberate
|
||||
architectural supersession (CTRL_PRIMARY_COMMITTING removed by ha-single-leader-role).**
|
||||
Counting the superseded pair as satisfied-by-replacement (their intent — gate on majority, observe
|
||||
degraded mode — is met more strongly), effective coverage is 33/33. No CRITICAL or HIGH functional
|
||||
gap remains.
|
||||
|
||||
## Residual / Deferred Items (all LOW, none blocking)
|
||||
|
||||
1. **Documentation drift (HIGH-doc, LOW-code).** requirements.md Req 6 + design.md still describe
|
||||
`CTRL_PRIMARY_COMMITTING`, `committing_timeout_ms`, and the 60s degraded-mode timeout, none of
|
||||
which exist in code. The Introduction (requirements.md:5-7) and Adjacent-expectations also
|
||||
reference COMMITTING. A reader of the spec will mis-model the live FSM. Code is correct; the
|
||||
prose is stale.
|
||||
2. **Req 5.2 "configurable" batch size** is compile-time (`#define`), not runtime config. Minor;
|
||||
no operator has needed to tune it. Defensible as "configurable at build."
|
||||
3. **`ha_compute_oldest_wal_lsn`** is an `opendir`/`readdir` scan per heartbeat reply per standby
|
||||
in vtask context (config_repl.c:1946). Performance-only; fine at current cluster sizes. The
|
||||
2026-06-09 caching suggestion (Gap 3.4) was never actioned and remains optional.
|
||||
4. **No covering tests** flagged by codegraph for `ha_process_standby_state`,
|
||||
`ha_evaluate_repl_action`, `ha_dispatch_repl_action_for_standby`, `ha_send_snapshot_to_standby`
|
||||
at the unit level (decision-loop logic is exercised via hooks in test_ha_role.c /
|
||||
test_config_repl.c, and convergence via test_convergence_verification.c). Coverage is
|
||||
integration-weighted, not absent — but the pure decision functions lack direct unit pins.
|
||||
5. **E2E entity-state assertion** (2026-06-09 Recommendation 5): tests 6.5–6.7 historically
|
||||
asserted LSN convergence only. With the data plane now real this is the right place to add a
|
||||
"submitted job visible on promoted standby" assertion to guard against future regressions to
|
||||
header-only application. Not done; LOW (data plane verified by unit round-trips per spec.json
|
||||
note).
|
||||
|
||||
## Effort / Risk for Open Work
|
||||
|
||||
| Item | Option | Effort | Risk |
|
||||
|---|---|---|---|
|
||||
| 1 — Spec/design doc reconciliation (remove COMMITTING; document own-term gate + write-quorum + safe_committed_lsn as the actual Req 6 mechanism) | **A (recommended):** update requirements.md Req 6 + Introduction + design.md to describe the superseding mechanism and cross-link `ha-single-leader-role` / `ha-commit-durability`. **B:** leave a one-line "SUPERSEDED" banner pointing to those specs. | A: S (0.5–1d) · B: XS (<0.5d) | Low (docs only) |
|
||||
| 2 — Promote batch limits to `ha_config_t` | A: plumb two fields. B: leave as build-time. | S (0.5d) | Low |
|
||||
| 3 — Cache `oldest_wal_lsn` | A: cache + invalidate on segment GC. B: defer (current scale fine). | S (0.5d) | Low |
|
||||
| 4 — Direct unit tests for decision functions | A: add `ha_evaluate_repl_action` truth-table + `ha_process_standby_state` convergence-flag tests. B: rely on hook/integration coverage. | S (1d) | Low |
|
||||
| 5 — E2E entity-state assertion | A: extend test 6.7 to assert a submitted job is present post-failover. B: rely on unit payload round-trips. | S (0.5–1d) | Low |
|
||||
|
||||
**Recommendation:** the spec is functionally complete and may remain `implemented`. The only item
|
||||
worth scheduling is **Item 1 (doc reconciliation)** — the requirements/design now misdescribe the
|
||||
live FSM after `ha-single-leader-role`, which will mislead future readers and any
|
||||
`/kiro-verify-completion` run that checks prose against code. Items 2–5 are optional hardening.
|
||||
|
|
|
|||
|
|
@ -133,3 +133,61 @@ Documentation:
|
|||
- **Build vs adopt**: nothing new is built beyond `WAL_NOOP`; the gate, barrier, classification guard, broadcast, and catch-up are all consumed from upstream specs/existing code. Deletions dominate (net-negative LOC in ha.c).
|
||||
- **Simplification**: `ha_promote_to_primary` survives only as the legacy/test-path wrapper around the shared helper; no compatibility shim for COMMITTING is kept anywhere (tests asserting it are rewritten or deleted, not stubbed).
|
||||
- Review gate: all 8 requirement IDs traced to components; no orphan components; boundary↔file alignment verified against the census table.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
**Re-audit context**: roadmap.md lists this spec as DONE, but `spec.json.phase` = `tasks-generated` and `tasks.md` shows 6 unchecked `[ ]` against 11 checked `[x]`. This pass determines whether the 6 open items are genuine remaining work or non-work artifacts, and re-verifies every EARS requirement against the live working tree (not the 2026-06-11 pre-implementation anchors in §3).
|
||||
|
||||
### A. Verification method
|
||||
- Repo-wide grep for the deleted-symbol set (`CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence`), excluding `.git`/`.kiro`/`.wolf`/`.codegraph`/`builddir`.
|
||||
- Docs/steering grep over the task-6.1 target set.
|
||||
- codegraph_explore of the promotion path, dispatch gate, role predicates, legacy path; direct grep of `ha.c`, `ha.h`, `wal_types.h`, `persist.c`, `meson.build`.
|
||||
- git history of `.kiro/specs/ha-single-leader-role/` and the controller tree.
|
||||
|
||||
### B. Requirement-to-Asset map (verified against live code, 2026-06-18)
|
||||
|
||||
| Req | EARS criterion | Asset (file:line) | Verdict |
|
||||
|-----|----------------|-------------------|---------|
|
||||
| 1.1 | Election winner → PRIMARY directly, no transitional role | `raft_cb_become_leader` direct store `ha.c:1034` (`→ CTRL_PRIMARY` log `ha.c:1036`); legacy `ha_elect_evaluate_and_signal` direct store `ha.c:~2919` (`→ CTRL_PRIMARY` log) | **MET** |
|
||||
| 1.2 | Single leader role; zero residual refs (code+tests+docs) | `ctrl_role_t` = 4 values `ha.h:60-64` (no `CTRL_PRIMARY_COMMITTING`); repo-wide deleted-symbol grep = **0**; docs/steering grep = **0** | **MET** |
|
||||
| 1.3 | Live primary processes via barrier, never redirects | `ha_dispatch_gate` PRIMARY branch `ha.c:2067` (no COMMITTING case); `ha_should_redirect` PRIMARY→0 unchanged; barrier-rollout fact allowlist (notify/update/setpreason/clrpreason) preserved `ha.c:~2099-2105` | **MET** |
|
||||
| 1.4 | Primary self-reports as primary pre-own-term-commit | `ha_is_primary` `ha.c:1466-1470` tests PRIMARY only; promotion stores PRIMARY before/without waiting on noop commit | **MET** |
|
||||
| 1.5 | No wall-clock leadership change | `committing_timeout_fn`/`_for_test`/`committing_timeout_ctx_t` deleted (grep 0); no watchdog armed in either promotion path; only `ha_arm_own_term_gate` (no timeout) on the tail | **MET** |
|
||||
| 2.1 | No confirmation before an own-term entry commits; entry created at promotion | `WAL_NOOP=40` `wal_types.h:57`; `ha_arm_own_term_gate` `ha.c:2259-2286` appends WAL_NOOP → `commit_submit(RETAIN)` → `commit_set_writable_lsn(noop_lsn)`; apply no-op `persist.c:1381`; replay no-op `persist.c:2037` | **MET** |
|
||||
| 2.2 | Pre-own-term request: barrier or NOT_READY only, no third path | Gate COMMITTING-reject branch gone; PRIMARY branch returns 0 (→ barrier) or `-XCE_ERR_NOT_READY` (`ha.c:~2105`); no promotion-side completion path exists | **MET** |
|
||||
| 2.3 | No timeout bypassing the own-term gate | watchdog/ack/convergence machinery deleted without replacement (grep 0); gate is `committed_lsn >= writable_lsn` only, engine-owned | **MET** |
|
||||
|
||||
**Requirements met: 8 / 8.**
|
||||
|
||||
Supporting structural confirmations:
|
||||
- `ha_register_persist_hooks` (`ha.c:2223`) + `ha_arm_own_term_gate` (`ha.c:2259`) are the shared promotion sequence; both the raft path (`ha.c:1122`,`1134`) and `ha_promote_to_primary` (`ha.c:2305`,`2312`) and the legacy path (`ha.c:~2920`) call them — single promotion flavor, no divergence.
|
||||
- Raft path does NOT call `ha_do_promote` (term synced at entry); `ha_promote_to_primary` keeps `ha_do_promote` (owns term) — double-increment wart resolved per research §4 finding 2.
|
||||
- Test disposition exact-match to plan: `test_ha_role_lifecycle.c` present + registered (`meson.build:94`); `test_ha_repl_ack.c` and `test_ha_evaluate_committing.c` deleted (absent on disk, absent from meson).
|
||||
- barrier-rollout gate tests intact and asserting the preserved PRIMARY fact allowlist (`test_commit_barrier_rollout.c` fact_gate_* cases).
|
||||
|
||||
### C. Open-task classification (the 6 unchecked `[ ]`)
|
||||
|
||||
`tasks.md` uses a two-level checkbox scheme: integer rows (`- [ ] 1.`…`6.`) are **phase group headers**; the actual work lives in decimal sub-tasks (`- [x] 1.1`, `1.2`, `2.1`, …). All 11 decimal sub-tasks are `[x]`. The 6 unchecked items are EXACTLY and ONLY the 6 group headers.
|
||||
|
||||
| `[ ]` item | Type | Evidence |
|
||||
|---|---|---|
|
||||
| 1. WAL_NOOP entry type | **GROUP-HEADER-ARTIFACT** | sub-tasks 1.1 (`6df1283`), 1.2 (`e6abd7c`) both `[x]` + committed; WAL_NOOP live |
|
||||
| 2. Promotion rewrite | **GROUP-HEADER-ARTIFACT** | 2.1 (`b7a69af`), 2.2 (`c089fb6`) `[x]` + committed; direct-PRIMARY + gate live |
|
||||
| 3. Legacy path symmetry | **GROUP-HEADER-ARTIFACT** | 3.1 (`b7b951b`) `[x]` + committed; legacy path symmetric, live |
|
||||
| 4. Machinery deletions | **GROUP-HEADER-ARTIFACT** | 4.1 (`a5bf861`), 4.2 (`71ce41f`) `[x]` + committed; grep 0 |
|
||||
| 5. Fixture sweep + symbol removal | **GROUP-HEADER-ARTIFACT** | 5.1 (`7f974e7`), 5.2 (`93dfc02`) `[x]` + committed; enum collapsed |
|
||||
| 6. Docs sync + grep verification | **GROUP-HEADER-ARTIFACT** | 6.1 (`dbc6316`), 6.2 (`e558a62`) `[x]` + committed; docs grep 0, ASAN-green evidence in commit |
|
||||
|
||||
**Genuine gaps: 0. Deferred: 0. Group-header artifacts: 6/6.**
|
||||
|
||||
The headers are unchecked purely because the kiro task generator does not auto-check parent rows when all children complete; they carry no independent deliverable. Task 6.2's commit (`e558a62`) explicitly records the final zero-match grep gate + ASAN regression green — the spec's own completion gate, which this re-audit independently reproduces (grep = 0 across code and docs).
|
||||
|
||||
### D. Options for the open items (no genuine engineering work exists)
|
||||
|
||||
- **Option A (recommended) — Administrative close-out.** Tick the 6 group-header checkboxes and flip `spec.json.phase` to `implemented`/`completed`. Effort: trivial (~2 min, doc-only). Risk: none. Rationale: every child task is committed and independently verified; roadmap already reflects DONE.
|
||||
- **Option B — Leave as-is.** Accept the cosmetic header/`spec.json` lag. Effort: zero. Risk: low-but-real — future `/kiro-spec-status` will keep reporting 6 "open" tasks and a `tasks-generated` phase, inviting repeat audits like this one (false-signal cost).
|
||||
- **Option C — Re-run a clean regression as fresh evidence.** `/xce-build controller` (debug + ASAN + suite) to re-confirm green on the current tree, then close out as Option A. Effort: one build cycle (note research §Implementation-Notes 5.1: meson 300s cap is borderline — prefer the direct binary run; a `Timeout: 1` is a wrapper artifact, not a failure). Risk: none; pure belt-and-suspenders. Recommended only if a green build artifact is required for sign-off beyond the existing task-6.2 commit evidence.
|
||||
|
||||
**Effort/Risk for genuine work: N/A — there is no genuine remaining work.** Closing recommendation: **Option A**.
|
||||
|
|
|
|||
62
.kiro/specs/ha-term-safety/TLA-gap-analysis.md
Normal file
62
.kiro/specs/ha-term-safety/TLA-gap-analysis.md
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
# Gap Analysis: XCE HA implementation vs `raft/raft.tla`
|
||||
|
||||
Reference spec: `raft/raft.tla` (Diego Ongaro's canonical Raft TLA⁺, 471 lines).
|
||||
Implementation: `src/lib/raft/raft.c` (election library) + the controller log/replication/commit layer (`persist.c`, `repl.c`, `ha.c`, `ha_convergence.c`, `config_repl.c`).
|
||||
Date: 2026-06-21. Grounds the `ha-term-safety` spec and bug-851.
|
||||
|
||||
## Headline
|
||||
|
||||
`raft.c` implements **only the election half** of the TLA⁺ spec — `currentTerm`, `votedFor`, `state`, `RequestVote`, `BecomeLeader`, `UpdateTerm` (on vote RPCs), `Timeout`. It has **no `log`, no `AppendEntries`, no `AdvanceCommitIndex`, no `commitIndex`**. The replicated log and its replication/commit/reconcile are implemented **separately** in the controller (WAL = `log`, catch-up streaming = `AppendEntries`, commit barrier = `AdvanceCommitIndex`, `ha_convergence.c` = conflict reconcile) and **do not follow the TLA⁺ log protocol**. Every bug-851 defect lives in that split: the election module is faithful, but the log layer omits the term-coupling the TLA⁺ spec uses to guarantee Log Matching and Leader Completeness.
|
||||
|
||||
## Action-by-action conformance
|
||||
|
||||
| TLA⁺ action / invariant | Implementation site | Status |
|
||||
|---|---|---|
|
||||
| `Timeout` (term++, clear vote, self-vote) | `raft_start_election` (raft.c:61) — `s->term++`, `voted_for=self`, persist before RPC | **Conformant** |
|
||||
| `RequestVote` freshness `logOk` = `mlastLogTerm > LastTerm ∨ (= ∧ mlastLogIndex ≥ Len)` | `on_vote_eligible` gate (raft.c:171) → controller LSN floor | **GAP G6** — freshness is **LSN-only**; never compares `lastLogTerm` (the WAL has no usable term) |
|
||||
| `HandleRequestVoteRequest` grant rules (≤ one vote/term, persist) | `raft_on_vote_request` (raft.c:110): stale-term deny, dup-vote deny, persist-before-grant | **Conformant** (Election Safety primitives correct) |
|
||||
| `BecomeLeader` (votesGranted ∈ Quorum) | `raft_on_vote_reply` step 5 (raft.c:373) `votes_granted ≥ n/2+1` | **Conformant** |
|
||||
| `UpdateTerm` — **any** RPC with higher term advances `currentTerm`, steps down | `raft_on_vote_request` step 2 + `raft_on_vote_reply` step 1 (vote RPCs **only**) | **GAP G5** — term adoption is wired to **vote RPCs only**; the AppendEntries/heartbeat-equivalent path does **not** route the leader's term into raft state |
|
||||
| `ClientRequest` — append `[term ↦ currentTerm, value]` | `persist_wal_write*` (persist.c) — `hdr.term = p->term`, `p->term` never set | **GAP G1** — entry term is **always 0**; the log carries no term |
|
||||
| `AppendEntries` / `HandleAppendEntriesRequest` `logOk` = `mprevLogTerm = log[prevLogIndex].term` | WAL catch-up stream + `persist_wal_write_replicated` contiguity guard (bug-850) | **GAP G2** — replication matches **LSN contiguity only**; no `prevLogTerm` check |
|
||||
| `HandleAppendEntriesRequest` conflict branch — delete entry whose `term ≠` leader's + all after | `ha_convergence.c` rejoin reconcile (truncates above committed boundary) | **GAP G3** — truncates by committed-boundary heuristic, **not by `(term,lsn)` conflict**; cannot detect same-LSN/different-term divergence |
|
||||
| `AdvanceCommitIndex` — commit `Max(agreeIndexes)` **only if** `log[Max].term = currentTerm` (Figure-8 safety) | commit barrier + `ha_compute_safe_committed_lsn` | **GAP G4** — current-term check is unenforceable (`term=0`); the own-term `WAL_NOOP` gate is a **partial** mitigation, but the safe-committed computation itself is term-blind |
|
||||
| `Restart` — keep `currentTerm`, `votedFor`, `log` | recover WAL + `ha-term.bin`; `raft_init` restores `(term, voted_for, voted_for_term)` | **Mostly conformant**, but feeds G5: a restarted node's raft term is only as fresh as what reached `ha-term.bin`; with G5 it never advanced from heartbeats, so it recovers **stale** |
|
||||
| Election Safety (≤ one leader/term) | emergent from the above | **VIOLATED in practice** — see below |
|
||||
| Log Matching (same idx+term ⇒ identical prefix) | emergent from the log layer | **VIOLATED** (bug-851) — G1/G2/G3 |
|
||||
| Leader Completeness (committed entry survives in all future leaders) | emergent | **VIOLATED** (bug-851) — G4 commits term-blind |
|
||||
| State Machine Safety (same idx applied ⇒ same command) | emergent | **VIOLATED** (bug-851 repro: divergent jobs at same committed LSN) |
|
||||
|
||||
Extensions beyond the TLA⁺ spec (not gaps): PreVote (`raft_begin_pre_vote`, disruption avoidance), single-node short-circuit, even-cluster-size rejection, priority/jitter staggering.
|
||||
|
||||
## How the gaps compose into bug-851 (observed)
|
||||
|
||||
1. **G5** — a standby never adopts the leader's term from heartbeats/replication (confirmed live: rejoined ctrl-1 reported `term: 2` while serving under a term-3 primary). So its raft `currentTerm` lags.
|
||||
2. On failover that lagged node runs `raft_start_election`, computing the new term as **stale + 1** → it **reuses a term** a prior, now-dead leader already used (observed: term 3 led by ctrl-3 then ctrl-1; term 4 by ctrl-2 then ctrl-3). Election Safety is bypassed not by a double-vote in one live term, but by **term-number reuse across the restart churn** that the vote-only term plumbing can't see.
|
||||
3. **G1** — each such leader appends its own-term no-op + jobs with `hdr.term = 0` at overlapping LSNs (both wrote `lsn=45`).
|
||||
4. **G2/G3** — replication/reconcile compare by LSN only, so `(term=3, lsn=45)` and `(term=4, lsn=45)` look identical (both type+term 0) and the divergent suffix is never truncated.
|
||||
5. **G4** — `committed` advances term-blind "from 1 granted voter" over entries a majority never held.
|
||||
|
||||
Result: all members report `committedLsn=54` over divergent content at LSN 45–54 — a silent committed-prefix divergence (State Machine Safety violation).
|
||||
|
||||
## Configuration is a second, parallel violation
|
||||
|
||||
Config commits through a **separate** index (`committed_config_tag`) with **no term** and tag-based anti-entropy (`config_repl.c`) — it is entirely outside both the TLA⁺ log and `raft.c`. It is exposed to the same term-reuse divergence, plus a state-materialization hazard: a config upgrade does config-only push + restart + **per-node WAL replay under the new config**, with no snapshot to pin an identical post-upgrade state (the `lost_and_found` re-homing is the tell that replay is config-dependent). The TLA⁺ spec assumes config changes are log entries; XCE's are neither log entries nor snapshot-anchored.
|
||||
|
||||
## Mapping gaps → `ha-term-safety` requirements
|
||||
|
||||
| Gap | Requirement that closes it |
|
||||
|---|---|
|
||||
| G1 entry term always 0 | R1 (per-entry WAL term recording) |
|
||||
| G5 term not adopted on replication path; term reuse | R2 (monotonic, durably-established term) |
|
||||
| G6 vote freshness LSN-only | R2.4 + R1 (term available to vote gate) |
|
||||
| G2 no `prevLogTerm` match | R3.1 (`(term,lsn)` comparison) |
|
||||
| G3 no conflict truncation | R3.2 (truncate divergent `(term,lsn)` suffix) |
|
||||
| G4 commit term-blind | R4.1/R4.5 (commit only `(term,lsn)`-quorum-corroborated; own-term) |
|
||||
| Config outside log / no term | R7 (config marker in unified log) |
|
||||
| Config-upgrade state re-derive | R8 (full snapshot at change boundary) |
|
||||
| All of the above | R6 (chaotic-failover regression test) |
|
||||
|
||||
## Note on scope vs `raft.c`
|
||||
|
||||
**DECISION (2026-06-21):** rather than patch the gaps in the controller's split architecture, the chosen direction is to **give `raft.c` a real log and route AppendEntries/commit through it — strict `raft.tla` conformance.** This is the `Raft TLA+ Conformance` series on the roadmap (`raft-log-replication` → `raft-commit-safety` → `raft-sim-conformance`, specs generated 2026-06-21). The library owns the protocol decisions (term on entries, `UpdateTerm` on every RPC, `(term,index)` matching+truncation, current-term commit, `(lastLogTerm,lastLogIndex)` freshness); the controller becomes a **thin callback adapter** (WAL = log, repl = transport, the sync commit barrier triggered via `on_commit_advanced`). `ha-term-safety` is reframed accordingly: its protocol invariants R1–R6 are **realized by the library specs** (not re-implemented in `ha_convergence.c`/`ha_compute_safe_committed_lsn`), and it retains only the controller-side wiring, the config marker + upgrade snapshot (R7/R8), and the live chaotic-failover regression. When `ha-term-safety` design is generated, map each requirement onto the library callbacks rather than controller-internal logic.
|
||||
443
.kiro/specs/ha-term-safety/design.md
Normal file
443
.kiro/specs/ha-term-safety/design.md
Normal file
|
|
@ -0,0 +1,443 @@
|
|||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose**: `ha-term-safety` is the **controller-side integration + config + live-regression umbrella** for the Raft TLA+ Conformance library series. It wires the XCE controller as a **thin callback adapter** to the protocol that `raft-log-replication`, `raft-commit-safety`, and `raft-sim-conformance` implement in `src/lib/raft/raft.c`, closing bug-851 (silent committed-prefix divergence under chaotic failover) **without re-implementing any Raft protocol logic in the controller**. (The related configuration-upgrade materialized-state divergence, bug-857, is descoped to `config-upgrade-snapshot`.)
|
||||
|
||||
**Users**: The XCE HA controller cluster (as Raft nodes), operators relying on "acknowledged ⇒ never lost across any failover pattern", and maintainers needing a live regression that reproduces and fences bug-851.
|
||||
|
||||
**Impact**: Today the replicated log, term propagation, reconcile, and commit are hand-rolled across `persist.c`/`repl.c`/`ha.c`/`ha_convergence.c`/`config_repl.c` and match by LSN only (term-blind). After this feature the controller **delegates** those decisions to the library through five new callbacks and three new entry points, **stamps the real election term** into the existing 40-byte WAL header, **derives committed config from an in-log `(term,lsn)` marker** instead of a separate index, and ships a live chaotic-multi-kill regression. Configuration **bytes** continue to move through the existing out-of-band config-push transport (`OPCode_replConfigPush`), hash-verified against the committed marker before activation. The Raft protocol itself (logOk, `(term,index)` truncation, current-term commit, freshness) is **not** in this spec — it is realized by the three library specs.
|
||||
|
||||
### Goals
|
||||
- Wire the controller adapter to the upstream callbacks **verbatim** (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`), stamping `wal_entry_hdr_t.term` from the library's term (R1) and routing incoming heartbeat/replication through the library so term adoption happens on every RPC (R2/R3).
|
||||
- Replace the term-blind safe-commit (`ha_compute_safe_committed_lsn`) and committed-boundary reconcile (`ha_rejoin_reconcile`) with the library's `commit_index` (via `on_commit_advanced`) and `(term,lsn)` truncation (via `on_truncate_from`), **preserving** the synchronous commit barrier + persist-before-ACK durability and the clean-failover behaviors of bugs 848/849/850 (R4/R5).
|
||||
- Order configuration changes through the same `(term,lsn)` log as a lightweight `WAL_CONFIG_MARKER` (version_tag + sha256, no bytes); derive committed config from the latest committed marker; deliver the configuration **bytes** via the existing out-of-band config-push transport, hash-verified against the committed marker before activation (R7).
|
||||
- Ship a live `xce_test` chaotic-multi-kill regression asserting cross-member `(term,lsn)`-fingerprinted committed-prefix identity (R6).
|
||||
|
||||
### Non-Goals
|
||||
- **The Raft protocol itself** — logOk/`prevLogTerm` match, `(term,index)` conflict truncation, the Figure-8 current-term commit rule, `(lastLogTerm,lastLogIndex)` freshness, `UpdateTerm`-on-any-RPC — owned by `raft-log-replication` + `raft-commit-safety`; consumed verbatim.
|
||||
- **The commit-barrier / durability mechanism** — the synchronous barrier, group-commit flusher, persist-before-ACK, reply gate, durable-floor freshness — owned by `ha-commit-durability`; this spec only changes the barrier's *input*, never its mechanism.
|
||||
- **The in-process conformance oracle** — Election Safety / Log Matching / Leader Completeness / State Machine Safety in `raft_sim.c` — owned by `raft-sim-conformance`; the R6 live regression complements it, it does not duplicate it.
|
||||
- **Configuration-upgrade materialized-state consistency** — establishing a full entity-bearing state snapshot at the change boundary so no member re-derives post-upgrade state (former R8) — is **descoped to the independent `config-upgrade-snapshot` spec**, which HARD-depends on `entity-bearing-snapshots`. It is **out of scope here**; this spec delivers the config marker (consensus) plus out-of-band config-push byte delivery, not a full upgrade snapshot.
|
||||
- Voting-set membership changes / joint consensus (`ha-config-bootstrap`); bugs 854/847/845/828 (tracked separately).
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
- **The controller callback-adapter bodies** implementing the five upstream callbacks over the WAL / repl transport / commit barrier (and an in-process route for incoming AppendEntries/heartbeat).
|
||||
- **`wal_entry_hdr_t.term` value stamping**: a `persist_set_term` seam so every append records the library's live term; reading `hdr.term` back on `replay_wal` and feeding it to `on_get_entry_term` + the library's restore (`initial_log_len`, last-log-term).
|
||||
- **`ha_convergence` delegation**: replacing the committed-boundary truncate with routing the leader's AppendEntries through `raft_on_append_entries_request` + `on_truncate_from`.
|
||||
- **Commit-source switch**: feeding the library's `commit_index` (via `on_commit_advanced`) into the existing synchronous barrier, retiring the term-blind `ha_compute_safe_committed_lsn` as the commit source.
|
||||
- **`WAL_CONFIG_MARKER` (R7)**: the new WAL entry type + append-via-`raft_log_append` + committed-config-derived-from-latest-committed-marker, retiring `committed_config_tag` as an authoritative index.
|
||||
- **Out-of-band config-push byte delivery + hash-verify-before-activate (R7)**: keying the existing config-push transport (`OPCode_replConfigPush`) to the committed marker's version_tag + sha256, verifying held bytes against the marker hash before activation, and fencing a member from serving under mismatched configuration.
|
||||
- **The live R6 regression**: the chaotic-multi-kill `xce_test` script + the `walcompare` cross-member `(term,lsn,pcrc)` divergence tool.
|
||||
|
||||
### Out of Boundary
|
||||
- All Raft protocol decisions (library: `raft-log-replication`, `raft-commit-safety`).
|
||||
- The commit barrier, durability/fsync, persist-before-ACK, durable-floor freshness mechanism (`ha-commit-durability`).
|
||||
- The conformance oracle / four-invariant checkers (`raft-sim-conformance`).
|
||||
- **Configuration-upgrade materialized-state consistency** — the full entity-bearing state snapshot at the change boundary (so no member re-derives post-upgrade state) and its `xzarray_freeze`/`thaw` + entity encode/decode payload — is **descoped to the independent `config-upgrade-snapshot` spec** (which HARD-depends on `entity-bearing-snapshots`). This spec ships the config marker (consensus) + out-of-band config-push byte delivery; it does **not** build the upgrade snapshot or entity serialization.
|
||||
- Membership/quorum transitions (`ha-config-bootstrap`); the on-disk WAL byte layout (unchanged — value fix only).
|
||||
|
||||
### Allowed Dependencies
|
||||
- `src/lib/raft/raft.{c,h}` after the three upstream specs land: the five callbacks + entry points + `raft_append_result_t` + `commit_index`/`next_index[]`/`match_index[]` + `(lastLogTerm,lastLogIndex)` freshness, consumed **verbatim**.
|
||||
- The existing controller infrastructure: `persist.c` WAL/recovery, `repl.c` transport, `commit.c` synchronous barrier + durability (`ha-commit-durability`), the out-of-band **config-push** transport (`OPCode_replConfigPush`, `config_repl.c`/`ha.c`), the ten already-wired election callbacks (`s_real_cbs`, `ha.c:1309-1320`).
|
||||
- The test infrastructure: `waldump.{c,h}`, `xce_test` script primitives, `/xce-deploy-ha`.
|
||||
|
||||
### Revalidation Triggers
|
||||
Re-check this design if any of the following change:
|
||||
- The signature/semantics of any upstream callback (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) or entry point (`raft_log_append`, `raft_send_append_entries`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`), including `raft_append_result_t`.
|
||||
- The 1-based-index ↔ WAL-LSN identity or the "term 0 == absent" convention.
|
||||
- The addition of `last_log_term` to `raft_on_vote_request`/`raft_on_pre_vote_request` (the controller must plumb it onto the wire and read it from `hdr.term` on recovery).
|
||||
- The `commit_index` restore convention (`initial_commit_index`) or `initial_log_len`.
|
||||
- The config-push transport contract (`OPCode_replConfigPush`) or the marker's `{version_tag, sha256}` key — the byte delivery is hash-verified against the committed marker before activation.
|
||||
- The synchronous commit barrier / persist-before-ACK contract (`ha-commit-durability`) — `on_commit_advanced` must continue to *trigger* it unchanged.
|
||||
- The split of the configuration-upgrade materialized-state work into `config-upgrade-snapshot` — if that spec changes how it consumes the `(term,lsn)` config marker, re-check the marker contract here.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
The controller already drives the **election** half of the library through ten callbacks wired in `s_real_cbs` (`ha.c:1309-1320`): vote/PreVote send, freshness gate (`raft_cb_vote_eligible`, `ha.c:299-360`), promotion/demotion (`raft_cb_become_leader`/`follower`, `ha.c:880-1242`), and term persistence (`raft_cb_persist` → `ha_persist_term`, `ha.c:1251-1294, 2441-2474`). The controller uses **only** `src/lib/raft/raft.c` (no legacy `src/controller/raft.c`). The **log** half is hand-rolled and term-blind:
|
||||
- Every WAL append stamps `hdr.term = p->term` (`persist.c:588/726/1040`) but `p->term` is **never assigned** → term 0 everywhere (G1). The 40-byte header already has the `term` field (`wal_types.h:29-39`) — a value fix, not a layout change.
|
||||
- Incoming heartbeats/replication never adopt a higher term (`ha_consume_heartbeat_committed` drops stale only, `ha.c:5013-5036`) (G5).
|
||||
- Reconcile truncates by committed-boundary, not `(term,lsn)` (`ha_rejoin_reconcile`, `ha_convergence.c:295-366`; `persist_truncate_wal_above` LSN-only, `persist.c:2455-2565`) (G3).
|
||||
- Safe commit is computed term-blind from voter LSNs (`ha_compute_safe_committed_lsn`, `ha.c:480-509`) (G4); config commits through a separate `committed_config_tag` with no term (`config_repl.c:941-979`), with the bytes pushed out-of-band via `OPCode_replConfigPush` — the marker (consensus) will replace the term-blind index, while the byte transport is reused as-is.
|
||||
|
||||
The durability layer (`ha-commit-durability`) is correct and must be preserved: `majority_candidate_locked` self-caps at `persist_durable_lsn` (`commit.c:79-138`), persist-before-ACK on standbys (`persist_wal_write_replicated`), the pre-promotion eligibility fence (`ha.c:933-951`), advertise-real-durable-floor freshness, and the contiguity guard (`persist.c:691-706`, bug-850).
|
||||
|
||||
This design follows the **same Ports-and-Adapters discipline already established by the election callbacks**: the library decides; the controller performs I/O.
|
||||
|
||||
### Architecture Pattern & Boundary Map
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Library["src/lib/raft/raft.c — PROTOCOL OWNER (NOT this spec)"]
|
||||
APP["raft_log_append (ClientRequest)"]
|
||||
AEREQ["raft_on_append_entries_request<br/>UpdateTerm + logOk + truncate"]
|
||||
AEREP["raft_on_append_entries_reply"]
|
||||
ADV["raft_advance_commit<br/>current-term Figure-8"]
|
||||
VOTE["vote freshness lastLogTerm,lastLogIndex"]
|
||||
end
|
||||
subgraph Adapter["src/controller — THIN ADAPTER (this spec)"]
|
||||
PE["on_persist_entry<br/>persist_set_term + WAL write"]
|
||||
GET["on_get_entry_term<br/>WAL index / replay read-back"]
|
||||
SEND["on_send_append_entries<br/>repl envelope prevLogTerm+entryTerm"]
|
||||
TRUNC["on_truncate_from<br/>persist_truncate_wal_above"]
|
||||
COMMIT["on_commit_advanced<br/>TRIGGER existing sync barrier"]
|
||||
ROUTE["heartbeat/repl intake -> raft_on_append_entries_request<br/>(replaces ha_consume_heartbeat + ha_rejoin_reconcile)"]
|
||||
end
|
||||
subgraph Config["Config marker + out-of-band byte delivery (this spec)"]
|
||||
MARK["WAL_CONFIG_MARKER<br/>version_tag + sha256 via raft_log_append"]
|
||||
CPUSH["config-push byte delivery<br/>OPCode_replConfigPush, hash-verify vs marker before activate"]
|
||||
end
|
||||
subgraph Preserved["PRESERVED policy (other specs)"]
|
||||
BARRIER["sync commit barrier + persist-before-ACK<br/>ha-commit-durability"]
|
||||
CFGXPORT["existing config-push transport<br/>OPCode_replConfigPush"]
|
||||
end
|
||||
subgraph Test["Live regression (this spec)"]
|
||||
WC["walcompare term,lsn,pcrc"]
|
||||
SCRIPT["xce_test chaotic multi-kill"]
|
||||
end
|
||||
APP --> PE
|
||||
AEREQ --> GET & PE & TRUNC
|
||||
ROUTE --> AEREQ
|
||||
AEREP --> SEND
|
||||
ADV --> COMMIT
|
||||
COMMIT --> BARRIER
|
||||
MARK --> APP
|
||||
CPUSH --> CFGXPORT
|
||||
SCRIPT --> WC
|
||||
```
|
||||
|
||||
**Architecture Integration**:
|
||||
- **Selected pattern**: thin callback adapter (Ports-and-Adapters), identical to the election-callback pattern already in `ha.c`. The library owns protocol; the controller owns I/O.
|
||||
- **Domain boundaries**: protocol decisions (append/truncate/commit/vote) in `raft.c`; durable I/O (WAL write, repl send, barrier, config-push transport) in the controller; the marker + out-of-band byte delivery in this spec; the configuration-upgrade materialized-state snapshot is descoped to `config-upgrade-snapshot`.
|
||||
- **Existing patterns preserved**: `s_real_cbs` callback wiring, persist-before-ACK, the synchronous barrier, the pre-promotion fence, the contiguity guard, "controller never self-terminates".
|
||||
- **Steering compliance**: `ha-async-io.md` (no blocking in iomgr/vtask; barrier on its own path), `persistence.md` (durable floor / acknowledged⇒durable preserved; never persist commit index), `error-handling.md` (status returns, fail-loud non-promotable fence, never `exit`/`abort`), `testing.md` (TDD; tests in `testsuite/` before C changes).
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice / Version | Role in Feature | Notes |
|
||||
|-------|------------------|-----------------|-------|
|
||||
| Protocol core | `src/lib/raft/raft.c` (gnu17, after 3 upstream specs) | Owns Raft log/commit/freshness | Consumed verbatim; not modified here |
|
||||
| Adapter / WAL | `src/controller/persist.c` + `wal_types.h` | `hdr.term` stamping, recovery read-back, truncate | 40-byte header unchanged; `persist_set_term` seam added |
|
||||
| Adapter / transport | `src/controller/repl.c`, `typemap.h` | carry `prevLogTerm` + entry term on AppendEntries | extend repl envelope fields |
|
||||
| Adapter / commit | `src/controller/commit.c` (`ha-commit-durability`) | barrier triggered by `on_commit_advanced` | mechanism unchanged |
|
||||
| Config | `src/controller/config_repl.c` + `wal_types.h` | `WAL_CONFIG_MARKER`, marker-derived committed config | retires `committed_config_tag` index |
|
||||
| Config bytes | existing config-push transport (`OPCode_replConfigPush`, `config_repl.c`/`ha.c`) | deliver bytes out-of-band; hash-verify vs marker before activate | transport reused as-is; no snapshot |
|
||||
| Test | `xce_test` + `waldump.{c,h}` + `/xce-deploy-ha` | live chaotic regression + `walcompare` | ~90% infra exists |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Modified Files
|
||||
- `src/include/wal_types.h` — add `WAL_CONFIG_MARKER = 50` to `wal_entry_type`; add the marker payload struct `wal_config_marker_t { uint64_t version_tag; uint8_t sha256[32]; }` (no config bytes). **No change to `wal_entry_hdr_t` (40 bytes preserved, R1.7).**
|
||||
- `src/include/persist.h` — declare `persist_set_term(persist_t *p, uint64_t term)`; declare recovery accessors that surface per-entry term + restored `log_len`/last-log-term for the library init.
|
||||
- `src/controller/persist.c` — implement `persist_set_term` (sets `p->term`); make `replay_wal` retain `hdr.term` per LSN (in-memory term index) so `on_get_entry_term` and library restore can read it; `persist_truncate_wal_above` reused unchanged by `on_truncate_from`.
|
||||
- `src/controller/repl.c` + `src/include/typemap.h` — extend the AppendEntries-carrying envelope (`wal_catchup_entry_t`/`capnp_repl_envelope_t`) to carry `prev_log_index`, `prev_log_term`, and the entry's `term` explicitly, so a standby persists the leader's term (not local `p->term`).
|
||||
- `src/controller/ha.c` — wire the five new callbacks into `s_real_cbs` (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`); add the heartbeat/replication intake that routes through `raft_on_append_entries_request` (replacing the term-blind `ha_consume_heartbeat_committed` term handling); drive `raft_log_append` on the primary append path; pass `last_log_term` on the vote wire and decode it on intake; retire `ha_compute_safe_committed_lsn` as the commit source (the library's `commit_index` drives the barrier via `on_commit_advanced`).
|
||||
- `src/controller/ha_convergence.c` — replace `ha_rejoin_reconcile`'s committed-boundary truncate with delegation: route the leader's AppendEntries through the library; `on_truncate_from` performs the truncate. Keep the non-promotable retry-fence behavior (R3.5/R5).
|
||||
- `src/controller/config_repl.c` — `do_upgrade_config_with_bytes` appends a `WAL_CONFIG_MARKER` via `raft_log_append` (replacing the `committed_config_tag` push/commit); derive committed config from the latest committed marker in the committed prefix; key the existing out-of-band config-push (`OPCode_replConfigPush`) to the marker's `{version_tag, sha256}`, hash-verify the held bytes against the committed marker before activating, and fence a member from serving under mismatched configuration (R7.5/R7.8–R7.10). The full entity-bearing upgrade snapshot is **out of scope** (`config-upgrade-snapshot`).
|
||||
- `src/controller/commit.c` — no mechanism change; `on_commit_advanced` calls into the existing `commit_advance_locked`/`commit_set_writable_lsn` path with the library-supplied `commit_index`.
|
||||
|
||||
### New Files
|
||||
- `src/controller/testsuite/walcompare.{c,h}` — cross-member WAL diff by `(lsn → term, pcrc)`; nonzero exit on committed-prefix divergence. Reuses the `waldump.c` reader/decoder (roadmap-planned tooling, `roadmap.md:55`).
|
||||
- `src/controller/testsuite/test_term_safety_adapter.c` — unit/integration tests for the adapter callbacks (term stamping, `on_get_entry_term` read-back, truncate delegation, `on_commit_advanced` barrier trigger, marker append/derive).
|
||||
- `src/tools/test-term-safety-chaos.sh` (or an `xce_test` script under `testsuite/`) — the R6 chaotic-multi-kill driver invoking `walcompare`.
|
||||
|
||||
> Each file keeps one responsibility: protocol stays in `raft.c` (untouched); `persist.c` = WAL I/O + term value; `ha.c`/`ha_convergence.c` = callback bodies + intake routing; `config_repl.c` = marker + out-of-band config-push byte delivery; `testsuite/` = regression. The File Structure Plan reflects the boundary: no protocol logic file is created or modified, and no upgrade-snapshot file is added (descoped to `config-upgrade-snapshot`).
|
||||
|
||||
## System Flows
|
||||
|
||||
### Replicated append → standby persist with the leader's term (R1, R2, R3)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant P as Primary library (raft.c)
|
||||
participant PA as Primary adapter (this spec)
|
||||
participant SA as Standby adapter (this spec)
|
||||
participant S as Standby library (raft.c)
|
||||
Note over P: raft_log_append(value) — stamps s->term
|
||||
P->>PA: on_persist_entry(idx, term, entry)
|
||||
PA->>PA: persist_set_term(p, term) then WAL write hdr.term=term
|
||||
Note over P: raft_send_append_entries(peer, commit_index)
|
||||
P->>PA: on_send_append_entries(peer, prevIdx, prevTerm, leaderCommit, entry+entryTerm)
|
||||
PA-->>SA: repl envelope (carries prevLogTerm + entryTerm)
|
||||
Note over S: raft_on_append_entries_request()<br/>UpdateTerm FIRST (G5) then logOk (G2)
|
||||
S->>SA: on_get_entry_term(prevIdx) %% logOk
|
||||
alt term conflict at idx (G3)
|
||||
S->>SA: on_truncate_from(idx) then on_persist_entry(idx, leader_term)
|
||||
else append
|
||||
S->>SA: on_persist_entry(idx, leader_term) %% standby persists LEADER term
|
||||
end
|
||||
Note over SA: persist-before-ACK preserved -> WalAck only on durable
|
||||
```
|
||||
|
||||
Flow-level decisions:
|
||||
- The standby persists the **leader's** entry term (carried explicitly on the envelope), closing the `persist_wal_write_replicated` "best-effort `p->term`" hazard. `persist_set_term` keeps `p->term` aligned with the library term for the local-append path.
|
||||
- Incoming heartbeats are 0-entry AppendEntries routed through the **same** `raft_on_append_entries_request`, so a higher-term heartbeat triggers `UpdateTerm` (G5) — the controller no longer needs `ha_consume_heartbeat_committed` to adopt term.
|
||||
- The contiguity guard (bug-850) and persist-before-ACK (ha-commit-durability) are unchanged and sit beneath `on_persist_entry`.
|
||||
|
||||
### Commit advance → existing barrier; config marker (R4, R7)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant L as Leader library (raft.c)
|
||||
participant A as Adapter (this spec)
|
||||
participant B as Sync commit barrier (ha-commit-durability, PRESERVED)
|
||||
Note over L: replication updated match_index[] (library)
|
||||
Note over L: raft_advance_commit() — current-term Figure-8 (G4)
|
||||
alt commit_index strictly advances
|
||||
L->>A: on_commit_advanced(commit_index)
|
||||
A->>B: feed safe commit position into existing barrier
|
||||
Note over B: majority self-cap at durable_lsn + persist-before-ACK<br/>+ own-term writable gate (UNCHANGED)
|
||||
else held (prev-term / no quorum)
|
||||
Note over L: no callback; barrier untouched
|
||||
end
|
||||
Note over A: committed config = latest WAL_CONFIG_MARKER<br/>within committed prefix (derived, not a separate index)
|
||||
```
|
||||
|
||||
Flow-level decisions:
|
||||
- `on_commit_advanced` **triggers** the barrier; it never performs durability itself. `ha_compute_safe_committed_lsn` is retired as the commit source (the library's current-term-gated `commit_index` replaces it), but the post-election truncate-to-safe at promotion and the durable self-cap remain (R5).
|
||||
- A config change is an ordinary committed log entry: the `WAL_CONFIG_MARKER` commits through the same barrier and reconciles through the same `(term,lsn)` matching (R7.1/R7.3/R7.6/R7.11). The committed config is the latest committed marker (R7.7).
|
||||
|
||||
### Configuration bytes — out-of-band config-push, hash-verified against the committed marker (R7.5, R7.8–R7.10)
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
M["WAL_CONFIG_MARKER commits at term,lsn<br/>(version_tag + sha256, no bytes)"] --> NEED{"local bytes match<br/>marker sha256?"}
|
||||
NEED -->|no| PUSH["obtain bytes via existing config-push<br/>OPCode_replConfigPush, keyed to version_tag"]
|
||||
PUSH --> VERIFY["verify obtained bytes sha256 == marker hash"]
|
||||
NEED -->|yes| VERIFY
|
||||
VERIFY -->|match| ACTIVATE["activate configuration"]
|
||||
VERIFY -->|mismatch| FENCE["serve-fence: keep member non-serving<br/>under new config; retry (R7.9)"]
|
||||
```
|
||||
|
||||
Flow-level decision: the marker (consensus) is the authority; the **bytes travel out-of-band on the existing config-push transport** (`OPCode_replConfigPush`), never in a WAL payload (R7.4). A member obtains and **hash-verifies the bytes against the committed marker before activating** (R7.8), fences itself on mismatch (R7.9), and on recovery activates the configuration referenced by the latest committed marker (R7.10). The full materialized-state **upgrade snapshot** (former R8) is **out of scope** — see `config-upgrade-snapshot`.
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Adapter component(s) | Library/upstream realization | Flow |
|
||||
|-------------|---------|----------------------|------------------------------|------|
|
||||
| 1.1, 1.2 | record live/promotion term in entry | `on_persist_entry` + `persist_set_term`; `raft_log_append` (incl. own-term NOOP) | library stamps `s->term` | Append flow |
|
||||
| 1.3, 1.4 | transmit + persist leader term on replication | `on_send_append_entries` carries entryTerm; standby `on_persist_entry` writes leader term | `raft_on_append_entries_request` | Append flow |
|
||||
| 1.5 | recovery reads back per-entry term | `replay_wal` term index; `on_get_entry_term`; restore `initial_log_len`/last-log-term | library init restore | — |
|
||||
| 1.6 | no zero/placeholder term while term valid | `persist_set_term` keeps `p->term` live | — | — |
|
||||
| 1.7 | preserve 40-byte WAL layout (value fix) | `wal_entry_hdr_t` unchanged | — | — |
|
||||
| 1.8 | legacy zero term = lowest-known for matching | `on_get_entry_term` maps legacy 0 → lowest-known | library `(term,index)` match tolerant | — |
|
||||
| 2.1, 2.2 | persist won term before appending; not committed until majority | `raft_cb_persist`→`ha_persist_term`; own-term `WAL_NOOP` gate (`commit_set_writable_lsn`) | `BecomeLeader` + commit rule | Commit flow |
|
||||
| 2.3, 2.4, 2.6 | no re-elect at stale term; adopt higher term on any RPC | heartbeat/repl intake → `raft_on_append_entries_request`; `on_persist` | `UpdateTerm` (G5) | Append flow |
|
||||
| 2.5 | at most one leader advances committed per term | library current-term commit + freshness | `raft-commit-safety` G4/G6 | Commit flow |
|
||||
| 3.1, 3.2 | reconcile by `(term,lsn)`; truncate divergent suffix | `ha_convergence` delegation → `on_truncate_from` | `raft_on_append_entries_request` (G3) | Append flow |
|
||||
| 3.3 | never truncate committed prefix | library truncates only above match point; `on_truncate_from` honors it | library | — |
|
||||
| 3.4 | reconciled prefix byte-identical to leader | standby persists leader entry+term | library | Append flow |
|
||||
| 3.5, 3.6 | non-promotable retry; no false committed | keep loud non-promotable fence; do not report divergent as committed | `ha-commit-durability` fence preserved | — |
|
||||
| 4.1, 4.5 | advance committed only on `(term,lsn)` quorum + current-term | `on_commit_advanced` feeds barrier with library `commit_index` | `raft_advance_commit` (G4) | Commit flow |
|
||||
| 4.2, 4.3, 4.4 | committed entry byte-identical/preserved across leadership | library Leader Completeness + barrier durability | library + `ha-commit-durability` | Commit flow |
|
||||
| 5.1 | clean single failover converges identically | unchanged promotion truncate-to-safe + library matching | `raft_cb_become_leader` | — |
|
||||
| 5.2 | recovery-on-promotion + advertise-real-LSN (848/849) | `raft_cb_become_leader` + `raft_cb_vote_eligible` unchanged | preserved | — |
|
||||
| 5.3 | contiguity guard, no forward gap (850) | `persist_wal_write_replicated` guard unchanged beneath `on_persist_entry` | preserved | — |
|
||||
| 5.4 | no added steady-state ack latency | barrier path untouched; `on_commit_advanced` only changes source | preserved | — |
|
||||
| 6.1, 6.2, 6.3 | chaotic regression asserts `(term,lsn,content)` prefix identity | `walcompare` + chaos script | — | — |
|
||||
| 6.4 | no acked-committed job absent post-run | `expect-job $jobid` per acked submit | — | — |
|
||||
| 6.5 | pre-fix repro fails, post-fix passes | negative-direction assertion (term=0 pre-fix) | — | — |
|
||||
| 6.6 | config changes interleaved → same committed config ref | `walcompare` checks committed marker tag+hash | R7 derive | Config flow |
|
||||
| 7.1–7.4, 7.6, 7.7, 7.11 | config as `(term,lsn)` marker, no bytes, term-stamped, reconciled, derived, majority-committed | `WAL_CONFIG_MARKER` via `raft_log_append`; marker-derived committed config | library log/commit/reconcile | Config marker flow |
|
||||
| 7.5, 7.8, 7.9, 7.10 | bytes via out-of-band config-push; verify hash before activate; fence on mismatch; activate from latest committed marker on recovery | config-push byte delivery (`OPCode_replConfigPush`) keyed to marker `{version_tag, sha256}` + hash-verify-before-activate + serve-fence | existing config-push transport | Config bytes flow |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|
||||
|-----------|-------|--------|--------------|--------------------------|-----------|
|
||||
| WAL term adapter | Adapter | Stamp/read-back the library term in `hdr.term` | 1.1–1.8 | library callbacks (P0), `persist.c` (P0) | Service, State |
|
||||
| AppendEntries intake/route | Adapter | Route heartbeat/repl through the library; term adoption | 2.3–2.6, 3.1–3.4 | `raft_on_append_entries_request` (P0), repl envelope (P0) | Service |
|
||||
| Reconcile delegation | Adapter | Replace committed-boundary truncate with `on_truncate_from` | 3.1–3.6 | library (P0), `persist_truncate_wal_above` (P0), fence (P1) | Service |
|
||||
| Commit-source switch | Adapter | Feed library `commit_index` into existing barrier | 4.1–4.5, 5.4 | `on_commit_advanced` (P0), `commit.c` barrier (P0) | Service |
|
||||
| Clean-failover preservation | Adapter | Keep 848/849/850 behaviors unchanged | 5.1–5.4 | promotion/vote callbacks, contiguity guard (P0) | Service |
|
||||
| Config marker | Adapter | `WAL_CONFIG_MARKER` in unified log; derive committed config | 7.1–7.4,7.6,7.7,7.11 | `raft_log_append` (P0), `config_repl.c` (P0) | Service, State |
|
||||
| Config byte delivery | Adapter | Out-of-band config-push keyed to marker; hash-verify-before-activate; serve-fence on mismatch | 7.5,7.8,7.9,7.10 | config-push transport `OPCode_replConfigPush` (P0), `config_repl.c` (P0) | Service |
|
||||
| Live regression | Test | Chaotic-multi-kill + `walcompare` | 6.1–6.6 | `xce_test`, `waldump`, `/xce-deploy-ha` (P0) | Test |
|
||||
|
||||
### Adapter Callback → Controller Implementation Site (the core seam table)
|
||||
|
||||
| Library callback / entry point (upstream) | Controller implementation site | Behavior |
|
||||
|---|---|---|
|
||||
| `on_persist_entry(ctx,index,term,entry,len)` | `ha.c` callback → `persist_set_term(p,term)` + `persist_wal_async`/`persist_wal_write_replicated` (`persist.c:588/726`) | Stamp `hdr.term=term` at LSN=index; persist-before-ACK preserved. |
|
||||
| `on_get_entry_term(ctx,index)->term\|0` | `ha.c` callback → in-memory WAL term index from `replay_wal`/live append | Return recorded `hdr.term` for LSN; legacy 0 → lowest-known (R1.8); 0 if absent. |
|
||||
| `on_send_append_entries(ctx,peer,prevIdx,prevTerm,leaderCommit,...)` | `ha.c` callback → `repl.c` envelope build (`typemap.h` extended fields) | Enqueue AppendEntries carrying `prevLogTerm` + entry `term`; `leaderCommit = commit_index`. |
|
||||
| `on_truncate_from(ctx,index)` | `ha.c` callback → `persist_truncate_wal_above(p,index-1)` (`persist.c:2455`) | Delete LSN≥index; replaces `ha_rejoin_reconcile` committed-boundary heuristic. |
|
||||
| `on_commit_advanced(ctx,commit_index)` | `ha.c` callback → `commit.c` existing barrier (`commit_advance_locked`/`commit_set_writable_lsn`) | Feed safe commit position; **trigger** barrier, do not replace it. |
|
||||
| `raft_log_append(s,entry,len,&idx)` | primary append path in `ha.c`/`config_repl.c` (incl. `WAL_CONFIG_MARKER`) | Library stamps `s->term`, assigns LSN; drives `on_persist_entry`. |
|
||||
| `raft_on_append_entries_request(...)` | heartbeat/repl intake in `ha.c` (replaces `ha_consume_heartbeat_committed` term handling) | Library does `UpdateTerm`+logOk+truncate/append via callbacks. |
|
||||
| `raft_on_append_entries_reply(...)` | repl ACK intake in `ha.c` | Library updates `match_index[]`/`next_index[]`. |
|
||||
| `raft_advance_commit(s)` | driven after repl ACK in `ha.c` | Library computes current-term-gated `commit_index`; may fire `on_commit_advanced`. |
|
||||
| `raft_on_vote_request/pre_vote_request(...,last_log_term,...)` | existing vote callbacks in `ha.c`, extended | Controller decodes candidate `last_log_term` from wire; reads own via `on_get_entry_term(log_len)`. |
|
||||
|
||||
### WAL term adapter
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Make `hdr.term` carry the library's election term on every entry and surface it on recovery |
|
||||
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `persist_set_term(persist_t *p, uint64_t term)` assigns `p->term` (the missing seam); called by `on_persist_entry` and on term adoption so no append stamps 0 while a valid term is in effect (R1.6). On-disk `wal_entry_hdr_t` is unchanged (40 bytes, R1.7).
|
||||
- `replay_wal` retains each `hdr.term` in an in-memory per-LSN index so `on_get_entry_term` answers from recovered state (R1.5); a legacy entry with term 0 maps to the lowest-known term for matching (R1.8).
|
||||
- Standbys persist the **leader's** entry term (carried on the envelope), not local `p->term` (R1.3/R1.4).
|
||||
|
||||
**Contracts**: Service [x] / State [x]
|
||||
|
||||
##### Service Interface
|
||||
```c
|
||||
/* Set the controller's live election term so subsequent WAL appends stamp it
|
||||
* into hdr.term (R1.1/R1.6). Called from on_persist_entry and on term adoption.
|
||||
* No on-disk layout change; value fix only (R1.7). */
|
||||
void persist_set_term(persist_t *p, uint64_t term);
|
||||
|
||||
/* Recovered per-entry term for an LSN (1-based == index), feeding on_get_entry_term
|
||||
* and the library's init restore. Returns 0 when absent; legacy-0 is mapped to the
|
||||
* lowest-known term by the adapter for (term,lsn) matching (R1.8). */
|
||||
uint64_t persist_entry_term(const persist_t *p, uint64_t lsn);
|
||||
```
|
||||
- Preconditions: `p` valid; `lsn` within `[1, current_lsn)`.
|
||||
- Postconditions: `on_get_entry_term(lsn)` returns the recorded term; appends stamp the live term.
|
||||
- Invariants: `hdr.term` never 0 while a valid term is in effect; 40-byte layout preserved.
|
||||
|
||||
### Config marker
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Order config changes through the unified `(term,lsn)` log as a lightweight marker |
|
||||
| Requirements | 7.1, 7.2, 7.3, 7.4, 7.6, 7.7, 7.11 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `do_upgrade_config_with_bytes` (`config_repl.c:1001-1163`) appends `WAL_CONFIG_MARKER` (`wal_config_marker_t = {version_tag, sha256[32]}`, **no bytes**, R7.2/R7.4) via `raft_log_append`, so it inherits term-stamping (R7.3), commit (R7.11), and `(term,lsn)` reconcile/truncate (R7.6).
|
||||
- Committed config is derived from the **latest committed marker within the committed prefix** (R7.7); the separate `committed_config_tag` index is retired as authoritative (it may remain as a derived cache).
|
||||
|
||||
**Contracts**: Service [x] / State [x]
|
||||
|
||||
##### Service Interface
|
||||
```c
|
||||
/* Append a config-change marker (version_tag + sha256, no config bytes) to the
|
||||
* unified replicated log via raft_log_append. Returns RAFT_OK / the append rc.
|
||||
* The config bytes travel out-of-band via config-push (R7.5), never the marker. */
|
||||
int config_append_marker(cluster_t *c, uint64_t version_tag,
|
||||
const uint8_t sha256[32], uint64_t *out_lsn);
|
||||
|
||||
/* Derive the committed configuration reference from the latest WAL_CONFIG_MARKER
|
||||
* at or below committed_lsn (R7.7). */
|
||||
int config_committed_marker(const cluster_t *c, uint64_t *version_tag,
|
||||
uint8_t sha256_out[32]);
|
||||
```
|
||||
- Preconditions: leader for append; valid `sha256`.
|
||||
- Postconditions: marker committed ⇒ committed config reference advances; reconcile truncates a divergent marker exactly like a job entry.
|
||||
- Invariants: config bytes never enter a WAL payload (R7.4).
|
||||
|
||||
### Config byte delivery
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Deliver config bytes out-of-band via the existing config-push transport; hash-verify against the committed marker before activation |
|
||||
| Requirements | 7.5, 7.8, 7.9, 7.10 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- Move the configuration bytes through the **existing out-of-band config-push transport** (`OPCode_replConfigPush`, `config_repl.c`/`ha.c`), keyed to the committed marker's `{version_tag, sha256}` — never through a WAL payload (R7.4/R7.5).
|
||||
- When a committed marker references bytes not yet held locally, obtain them via config-push and **verify the obtained bytes' sha256 against the marker hash before activating** (R7.8).
|
||||
- If a member's locally held bytes do not match the latest committed marker's hash, **keep the member from serving under the new configuration and keep retrying** rather than activating mismatched config (R7.9) — the "controller never self-terminates" loud non-promotable fence, never `exit`/`abort`.
|
||||
- On recovery, activate the configuration referenced by the **latest committed marker**, obtaining the bytes from out-of-band storage (R7.10).
|
||||
|
||||
**Contracts**: Service [x]
|
||||
|
||||
##### Service Interface
|
||||
```c
|
||||
/* Ensure the bytes for the latest committed config marker are held locally and
|
||||
* hash-verified against the marker, then activate. Obtains bytes via config-push
|
||||
* (OPCode_replConfigPush) when absent. Returns RAFT_OK on activate; a non-success
|
||||
* rc keeps the member fenced (non-serving under the new config) and retrying
|
||||
* (R7.8/R7.9/R7.10). */
|
||||
int config_activate_committed(cluster_t *c);
|
||||
```
|
||||
- Preconditions: a committed `WAL_CONFIG_MARKER` exists in the committed prefix.
|
||||
- Postconditions: on success the active config matches the marker's `{version_tag, sha256}`; on hash mismatch the member stays fenced and retries.
|
||||
- Invariants: bytes are never sourced from a WAL payload; activation only after hash-verify against the committed marker.
|
||||
|
||||
> **Note**: the full materialized-state **upgrade snapshot** (embed entities + config in one artifact, install on every member so no member re-derives post-upgrade state — former R8) is **out of scope** here and lives in the independent **`config-upgrade-snapshot`** spec, which HARD-depends on `entity-bearing-snapshots`. This spec delivers config bytes via the interim out-of-band config-push path that `config-upgrade-snapshot` later supersedes for the upgrade case.
|
||||
|
||||
### Live regression (R6)
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Reproduce bug-851 and assert cross-member `(term,lsn)` committed-prefix identity |
|
||||
| Requirements | 6.1, 6.2, 6.3, 6.4, 6.5, 6.6 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `walcompare <dirA> <dirB> ...` diffs members by `(lsn → term, pcrc)`; nonzero exit on any committed-prefix divergence (R6.1/R6.2/R6.3), reusing `waldump.c`'s decoder.
|
||||
- The chaos script drives repeated primary kills interleaved with `jobsub` + config upgrades via `/xce-deploy-ha` + `kill_node`/`start_node` + deterministic `--wal-sync-ms`, then asserts `expect-job $jobid` for every acked submit (R6.4) and `walcompare` identity incl. committed config marker (R6.6).
|
||||
- R6.5: on **pre-fix** code (term=0) the repro must fail (`walcompare` detects divergence / `expect-job` loss); on fixed code it passes.
|
||||
|
||||
**Contracts**: Test [x]
|
||||
|
||||
## Data Models
|
||||
|
||||
### Domain Model
|
||||
- **Log entry**: `(index==LSN, term, type, payload)` — the unified `(term,lsn)` log for both job state and config markers. Index 1-based == WAL LSN (library convention).
|
||||
- **Config-change marker**: `wal_config_marker_t = {uint64_t version_tag, uint8_t sha256[32]}` — a reference, never the bytes.
|
||||
- **Committed config**: derived value = the latest `WAL_CONFIG_MARKER` at or below `committed_lsn`.
|
||||
|
||||
### Logical Data Model
|
||||
- `wal_entry_type` gains `WAL_CONFIG_MARKER = 50` (between `WAL_NOOP=40` and `WAL_CHECKPOINT=100`).
|
||||
- `persist_t.term` becomes a written field via `persist_set_term`; a recovered per-LSN term index is added to `persist_t` (in-memory only).
|
||||
- The repl envelope (`wal_catchup_entry_t`/`capnp_repl_envelope_t`) gains explicit `prev_log_index`, `prev_log_term`, and entry `term` fields on the AppendEntries path.
|
||||
- **No on-disk WAL layout change** (R1.7): `wal_entry_hdr_t` stays 40 bytes with `term` at offset 0.
|
||||
|
||||
### Data Contracts & Integration
|
||||
- **AppendEntries wire**: carries `(prevLogIndex, prevLogTerm, leaderCommit, entry, entryTerm)` — the standby persists `entryTerm`, not local `p->term`.
|
||||
- **Vote wire**: carries `(lastLogTerm, lastLogIndex)`; the controller decodes `lastLogTerm` and passes it to the library; own value via `on_get_entry_term(log_len)`.
|
||||
- **Config bytes**: delivered out-of-band via config-push (`OPCode_replConfigPush`), keyed to the marker's `{version_tag, sha256}`, hash-verified against the committed marker before activation; never in a WAL payload. The full materialized-state upgrade snapshot is out of scope (`config-upgrade-snapshot`).
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Strategy
|
||||
The adapter follows "controller never self-terminates" (`error-handling.md`, `ha-async-io.md`): callbacks return status; on a non-recoverable condition the controller enters a **loud, non-promotable fence and keeps retrying** — it never `exit`/`abort`. Persist-before-ACK and the durable self-cap are unchanged.
|
||||
|
||||
### Error Categories and Responses
|
||||
- **Persist failure** (`on_persist_entry` rc≠0) → propagate; library does not advance the local log; standby does not ACK (persist-before-ACK preserved).
|
||||
- **Truncate failure** (`on_truncate_from` rc≠0) → report a non-success request result so the leader retries; the member stays non-promotable until reconciled (R3.5).
|
||||
- **Unreconcilable member** → loud permanent fence, non-promotable, keep retrying catch-up (R3.5/R5; never report false convergence, R3.6).
|
||||
- **Config hash mismatch** (R7.9) → keep the member from serving under the new config; retry; do not activate mismatched config (`config_activate_committed` returns non-success; member stays fenced).
|
||||
- **Config bytes absent for a committed marker** (R7.8) → obtain via config-push and hash-verify before activate; do not activate on a missing or mismatched payload.
|
||||
|
||||
### Monitoring
|
||||
Reuse `raft_cb_trace` and existing INFO/ERROR logging for: term-stamp, term-adopt-on-AE, conflict-truncate, commit-advance-trigger, config-marker-commit, config-byte-verify, and the loud fences.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
TDD-mandatory (`testing.md`): tests in `src/controller/testsuite/` precede C changes; gate on the ANSI-stripped `:FAIL:` count; run final durability/regression validation on a real fsync-able filesystem.
|
||||
|
||||
### Unit Tests (adapter callbacks, in `test_term_safety_adapter.c`)
|
||||
1. `on_persist_entry` stamps `hdr.term` from the library term (via `persist_set_term`) and `on_get_entry_term` reads it back; legacy-0 maps to lowest-known (R1.1–1.8).
|
||||
2. AppendEntries intake routes a higher-term heartbeat through `raft_on_append_entries_request`, the library adopts the term, and `ha_persist_term` records it (R2.3–2.6 / G5).
|
||||
3. `on_truncate_from` deletes the divergent suffix and the standby persists the leader entry+term; committed prefix untouched (R3.1–3.4 / G3).
|
||||
4. `on_commit_advanced` feeds the library `commit_index` into the existing barrier and the WAIT entry releases only on durable majority; `ha_compute_safe_committed_lsn` no longer the commit source (R4.1/R4.5, R5.4).
|
||||
5. `config_append_marker` appends `WAL_CONFIG_MARKER` (no bytes) and `config_committed_marker` derives committed config from the latest committed marker; a divergent marker truncates like a job entry (R7.1–7.7).
|
||||
|
||||
### Integration Tests
|
||||
1. Clean single failover converges to byte-identical WALs incl. terms, preserving recovery-on-promotion + advertise-real-LSN + contiguity (R5.1–5.3).
|
||||
2. Marker commit + reconcile under a forced leader change keeps every member on the same committed config reference (R6.6, R7.11).
|
||||
3. Config byte delivery: a committed marker whose bytes are obtained via config-push activates only after the bytes' sha256 matches the marker hash; an injected hash mismatch keeps the member fenced (non-serving under the new config) and retrying (R7.8/R7.9/R7.10).
|
||||
|
||||
### E2E / Live Regression (R6 — the headline)
|
||||
1. `xce_test` chaotic-multi-kill: 3-node cluster, repeated primary SIGKILL interleaved with `jobsub` and config upgrades under deterministic `--wal-sync-ms`; after the run `walcompare` asserts `(term,lsn,pcrc)` committed-prefix identity across all members and `expect-job $jobid` confirms no acked-committed job is absent (R6.1–6.4, R6.6).
|
||||
2. Negative direction: the same script on **pre-fix** code (term=0) reproduces bug-851 (committed-prefix divergence / job loss); on fixed code it passes (R6.5).
|
||||
|
||||
## Resolved Scope Note (descope of former R8)
|
||||
|
||||
The former Requirement 8 (full entity-bearing configuration-upgrade snapshot) was a **blocking item** in the prior draft of this design: it required a full materialized-state snapshot at the change boundary, which depends on **entity-bearing snapshots** that do not exist yet (today snapshots are a 16-byte LSN sentinel; the `xzarray_freeze`/`thaw` machinery is built but dormant — the unstarted `entity-bearing-snapshots` backlog item).
|
||||
|
||||
**Resolution (2026-06-21): the materialized-state-consistency-on-upgrade work is descoped into the independent [`config-upgrade-snapshot`](../config-upgrade-snapshot/brief.md) spec, which HARD-depends on `entity-bearing-snapshots`. It is out of scope here.**
|
||||
|
||||
What this spec ships now (no snapshot dependency):
|
||||
- R1–R6: per-entry term, monotonic term adoption, `(term,lsn)` reconcile, term-safe committed prefix, and the live chaotic-multi-kill regression — all realized on the library + existing transport.
|
||||
- R7: the config-change **marker** (consensus, `(term,lsn)`, no bytes) in the unified log, with configuration **bytes** delivered via the existing **out-of-band config-push transport** (`OPCode_replConfigPush`), hash-verified against the committed marker before activation. `config-upgrade-snapshot` later supersedes this interim byte delivery for the upgrade case once entity-bearing snapshots land.
|
||||
|
||||
Constraints preserved by the descope: config bytes are never carried in the WAL marker (R7.2/R7.4), and no member re-derives committed state from a stale separate index — the marker is the consensus authority.
|
||||
131
.kiro/specs/ha-term-safety/requirements.md
Normal file
131
.kiro/specs/ha-term-safety/requirements.md
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
This feature closes a Raft safety gap (bug-851) in the XCE HA controller cluster. Under chaotic controller failover (repeated loss of the primary), members can reach a state where every node reports the same `committedLsn` over a contiguous, CRC-intact WAL, yet the nodes hold **different entries at the same LSNs** — a silent committed-prefix divergence that loses acknowledged jobs. A clean single failover converges correctly today (validated end-to-end: bugs 848, 849, 850), so the defect is specific to leadership churn.
|
||||
|
||||
Root cause has three compounding parts, each addressed by a requirement area below:
|
||||
1. The election term is not monotonic across the cluster — a leader can act in a term that was never durably propagated to a majority, letting a restarted node re-elect at the same term, so two leaders share one term.
|
||||
2. The per-entry WAL term is never recorded (always `0`), so the WAL carries no Raft term information.
|
||||
3. Rejoin reconcile cannot perform `(term, lsn)` log-matching, so divergent same-LSN tails are never truncated and `committed` advances over entries a majority never held.
|
||||
|
||||
The objective is to restore the Raft Log Matching and Leader Completeness guarantees so that any entry reported `committed` is byte-identical on every member, including across arbitrary failover patterns — without regressing clean-failover behavior or the on-disk WAL format.
|
||||
|
||||
The same divergence class latently affects configuration, which today commits through a separate index (`committed_config_tag`) with no per-entry term. Rather than maintain two consistency domains, this feature unifies the **consensus** half: a configuration change is recorded as a **lightweight in-log marker** carrying only a reference to the configuration (version tag + content hash), ordered in the same `(term, lsn)` log as job state, so configuration inherits the same term-safety and reconcile. The **configuration bytes stay out of the WAL records** (the marker carries only a reference); the bytes continue to move through the existing **out-of-band config-push transport**, hash-verified against the committed marker before activation.
|
||||
|
||||
The remaining hazard — a configuration *upgrade* leaving members with divergent *materialized* state because XCE's state machine is interpreted against the configuration and each member re-derives post-upgrade state by replaying pre-upgrade WAL under the new configuration — requires a full entity-bearing state snapshot at the change boundary, which does not exist yet (snapshots are presently a 16-byte LSN sentinel). That work is **descoped to the independent `config-upgrade-snapshot` spec** (depends on entity-bearing snapshots) and is out of scope here.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**:
|
||||
- Recording the live election term in every WAL entry and carrying it through replication and recovery.
|
||||
- Making the election term monotonic and durably established before a new leader appends entries (no two leaders per term).
|
||||
- `(term, lsn)` log-matching during rejoin/reconcile, including truncation of divergent local suffixes.
|
||||
- Preserving the committed-prefix consistency invariant across arbitrary failover sequences.
|
||||
- **Unified ordering of configuration changes and job state through the same replicated log**, by recording each configuration change as a lightweight in-log marker that references the configuration (version tag + content hash) rather than carrying its bytes, so configuration inherits the same term-safety and reconcile guarantees. Configuration bytes continue to move through the existing out-of-band config-push transport, hash-verified against the committed marker.
|
||||
- A regression test that drives chaotic multi-kill failover and asserts cross-member committed-prefix identity (including configuration markers).
|
||||
- **Out of scope** (tracked separately):
|
||||
- **Configuration-upgrade materialized-state consistency** (full entity-bearing state snapshot at the change boundary so no member re-derives post-upgrade state) — **descoped to the independent `config-upgrade-snapshot` spec**, which depends on entity-bearing snapshots (`entity-bearing-snapshots` backlog item; today snapshots are a 16-byte LSN sentinel). This spec ships the config marker (consensus) but not the full-snapshot upgrade.
|
||||
- Client primary-discovery being positional (bug-854).
|
||||
- Catch-up payload drop across multiple WAL segments (bug-847).
|
||||
- Each job producing two `JOB_SUBMIT` WAL entries (bug-845).
|
||||
- Snapshot-based prefix backfill / empty-rejoin gap (bug-828, `ha-commit-durability`).
|
||||
- Changes to the controller voting set (adding/removing a controller peer): such membership transitions change the quorum itself and remain governed by `ha-config-bootstrap`; this spec orders the config-change marker uniformly but does not introduce joint-consensus quorum transitions.
|
||||
- **Adjacent expectations**:
|
||||
- Relies on the existing election/PreVote machinery (`ha-prevote`, `ha-single-leader-role`) and the commit barrier (`ha-commit-engine`) to provide term and quorum signals; this feature constrains how those signals are recorded and reconciled, and does not redesign the election protocol itself.
|
||||
- Relies on persistence recovery (`ha-commit-durability`) for the recovered durable floor; this feature adds the term dimension to what recovery and reconcile compare.
|
||||
- Carries configuration bytes via the existing out-of-band config-push transport, never via WAL entry payloads (the WAL holds only the `(term, lsn)` marker); the bytes are hash-verified against the committed marker before activation. The follow-on `config-upgrade-snapshot` spec later supersedes this byte delivery with a full entity-bearing upgrade snapshot once entity-bearing snapshots exist.
|
||||
- Realizes its protocol invariants (R1–R6) through the **raft library** callbacks delivered by the `Raft TLA+ Conformance` series (`raft-log-replication`, `raft-commit-safety`); this spec is the controller-side **adapter + config marker + live regression**, not a re-implementation of the Raft protocol.
|
||||
- Complements `ha-consistency-invariants` and `ha-convergence-recovery` by supplying the per-entry term those invariants assume already exists.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Per-entry WAL term recording
|
||||
|
||||
**Objective:** As an operator of an HA controller cluster, I want every WAL entry to record the election term under which it was created, so that entries from different terms are distinguishable and Raft log-matching is possible.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When the primary appends a WAL entry, the Controller shall record in that entry the value of the leader's current election term in effect at append time.
|
||||
2. When the primary appends its own-term no-op on promotion, the Controller shall record the newly established term in that no-op entry.
|
||||
3. When the primary replicates a WAL entry to a standby, the Controller shall transmit the entry's recorded term so that the standby persists the leader's term, not a substitute value.
|
||||
4. While applying a replicated entry, the standby shall persist the entry with the same term the leader recorded for that entry.
|
||||
5. When a member recovers its WAL at startup, the Controller shall read back each entry's recorded term and make it available to the reconcile and promotion logic.
|
||||
6. The Controller shall not record a fixed or zero placeholder term for entries created while a valid election term is in effect.
|
||||
7. The Controller shall preserve the existing on-disk WAL entry header size and field layout, treating the term as a value fix rather than a format change.
|
||||
8. Where a WAL entry predates this feature and carries a legacy zero term, the Controller shall treat it as belonging to the lowest-known term for matching purposes rather than failing recovery.
|
||||
|
||||
### Requirement 2: Monotonic, durably-established election term
|
||||
|
||||
**Objective:** As an operator, I want a node to become an acting leader only at a term that is strictly greater than any term already used in the cluster and durably established, so that two leaders never share a term.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a node wins an election, the Controller shall durably persist the won term locally before it appends any entry as leader.
|
||||
2. While a node has not propagated its won term to a majority of members, the Controller shall not treat any entry it appends in that term as committed.
|
||||
3. If a node recovers a stale election term after restart, then the Controller shall not begin or win a new election at a term less than or equal to any term it can observe from a reachable peer or its own persisted WAL.
|
||||
4. When a node initiates an election, the Controller shall choose a prospective term strictly greater than the highest term it has observed locally and from reachable peers.
|
||||
5. If two members would otherwise act as leader in the same term, then the Controller shall ensure at most one of them can advance committed in that term and the other steps down without having committed divergent entries.
|
||||
6. When a member observes a higher term from a peer than its own, the Controller shall adopt the higher term and abandon any leadership it holds at the lower term.
|
||||
|
||||
### Requirement 3: `(term, lsn)` log-matching reconcile
|
||||
|
||||
**Objective:** As an operator, I want a rejoining member to converge its WAL to the current leader's log, so that divergent tails created during churn do not persist.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a member rejoins and synchronizes with the current leader, the Controller shall compare its local WAL to the leader's by `(term, lsn)` pairs, not by `lsn` alone.
|
||||
2. If a member's local entry at a given LSN has a term that does not match the leader's entry at that LSN, then the Controller shall truncate that local entry and every entry after it before accepting the leader's entries.
|
||||
3. When the Controller truncates divergent local entries, it shall not truncate any entry that is part of the cluster's committed prefix.
|
||||
4. When reconcile completes, the member's WAL prefix up to the leader's committed point shall be byte-identical to the leader's.
|
||||
5. If a member cannot reconcile to a committed-consistent state with the leader, then the Controller shall keep that member non-promotable and continue retrying rather than reporting false convergence.
|
||||
6. While a member holds an unreconciled divergent suffix, the Controller shall not report those divergent entries as committed.
|
||||
|
||||
### Requirement 4: Committed-prefix consistency invariant
|
||||
|
||||
**Objective:** As a client and operator, I want any entry reported committed to be identical on every member, so that an acknowledged job is never lost or replaced after a failover.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The Controller shall only advance `committed` to an LSN whose entry is held with a matching `(term, lsn)` by a majority of members, applying the same rule to job-state entries and configuration-change markers.
|
||||
2. When any member reports an LSN as committed, the Controller shall ensure the entry at that LSN is byte-identical on every member that reports the same or a higher committed LSN.
|
||||
3. If an entry was acknowledged to a client as committed, then the Controller shall preserve that exact entry across all subsequent leadership changes.
|
||||
4. While leadership changes occur, the Controller shall never replace, reorder, or drop an entry within the established committed prefix.
|
||||
5. When a new leader computes its safe committed point from granted voters, the Controller shall not include an LSN whose `(term, lsn)` is not corroborated by the required quorum.
|
||||
|
||||
### Requirement 5: Clean-failover behavior preserved
|
||||
|
||||
**Objective:** As an operator, I want the validated clean-failover and recovery behavior to keep working unchanged, so that this fix introduces no regression.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a single clean failover occurs (one primary killed, one standby promoted, old primary restarted), the Controller shall converge all members to a byte-identical WAL as it does today.
|
||||
2. While a promoted standby recovers and advertises its recovered applied LSN, the Controller shall preserve the existing recovery-on-promotion and advertise-real-LSN behavior (bugs 849 and 848).
|
||||
3. When a standby falls behind and the primary streams catch-up entries, the Controller shall preserve the existing contiguity behavior that refuses a forward gap without dropping live entries (bug 850).
|
||||
4. The Controller shall not increase committed-acknowledgment latency for the clean, single-leader steady state beyond its current behavior.
|
||||
|
||||
### Requirement 6: Chaotic-failover regression verification
|
||||
|
||||
**Objective:** As a maintainer, I want an automated test that reproduces the chaotic-failover divergence, so that the fix is proven and protected against regression.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When the regression scenario drives repeated primary kills interleaved with job submissions, the test shall assert that every member's committed WAL prefix is byte-identical.
|
||||
2. When the test inspects member WALs, it shall compare entries by `(term, lsn)` and content so that a same-LSN divergence is detected, not masked by equal LSN ranges.
|
||||
3. If any member holds a divergent entry within the committed prefix, then the test shall fail with the diverging LSN and the per-member entry identities reported.
|
||||
4. When the regression scenario completes, the test shall confirm that no entry acknowledged as committed during the run is absent from any surviving member.
|
||||
5. The test shall reproduce the original bug-851 failure on the pre-fix code and pass on the fixed code.
|
||||
6. When the regression scenario interleaves configuration changes with the chaotic kills, the test shall assert that every member converges to the same committed configuration reference (version tag and content hash).
|
||||
|
||||
### Requirement 7: Unified configuration ordering with out-of-band configuration data
|
||||
|
||||
**Objective:** As an operator, I want configuration changes ordered and committed through the same replicated log and term machinery as job state, while the configuration bytes stay out of the WAL records (carried by the existing out-of-band configuration transport), so that configuration inherits term-safety and reconcile without bloating the log.
|
||||
|
||||
> **Scope note (descope 2026-06-21):** the full materialized-state consistency of a configuration *upgrade* — establishing a full state snapshot at the change boundary so no member re-derives post-upgrade state — is moved to the independent **`config-upgrade-snapshot`** spec, which depends on entity-bearing snapshots (a snapshot that carries entity state does not exist yet; today it is a 16-byte LSN sentinel). This requirement keeps the consensus half (the in-log `(term, lsn)` marker) and continues to deliver the configuration *bytes* through the existing out-of-band config-push transport, hash-verified against the marker.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a configuration change is committed, the Controller shall append a configuration-change marker to the same replicated log used for job state, ordered by the same `(term, lsn)` sequence.
|
||||
2. The Controller shall record in each configuration-change marker a reference to the configuration — its version tag and content hash — and shall not place the configuration bytes in the marker.
|
||||
3. The Controller shall record the live election term in each configuration-change marker, identically to job-state entries (Requirement 1).
|
||||
4. The Controller shall not store configuration bytes in WAL entry payloads.
|
||||
5. The Controller shall move the configuration bytes through the existing out-of-band configuration transport (config push), keyed to the marker's version tag and content hash, not through the WAL.
|
||||
6. When reconcile compares logs by `(term, lsn)`, the Controller shall truncate a divergent configuration-change marker exactly as it truncates a divergent job-state entry.
|
||||
7. The Controller shall derive the committed configuration from the latest configuration-change marker within the committed log prefix, rather than from a separate configuration commit index.
|
||||
8. While a configuration-change marker is committed but its referenced configuration bytes are not yet present locally, the Controller shall obtain the bytes through the out-of-band transport and verify them against the marker's content hash before activating the configuration.
|
||||
9. If a member's locally held configuration bytes do not match the content hash of the latest committed configuration marker, then the Controller shall keep that member from serving under the new configuration and continue retrying rather than activating mismatched configuration.
|
||||
10. When a member recovers, the Controller shall activate the configuration referenced by the latest committed configuration-change marker, obtaining the bytes from out-of-band storage.
|
||||
11. The Controller shall preserve the existing majority-acknowledgment requirement for a configuration change, now expressed as the configuration-change marker reaching the committed prefix.
|
||||
122
.kiro/specs/ha-term-safety/research.md
Normal file
122
.kiro/specs/ha-term-safety/research.md
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
# Research & Design Decisions
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `ha-term-safety`
|
||||
- **Discovery Scope**: Complex Integration (controller-side adapter + config marker + out-of-band config-push byte delivery + live regression umbrella for the Raft TLA+ Conformance library series; the configuration-upgrade materialized-state snapshot is descoped to `config-upgrade-snapshot`).
|
||||
- **Key Findings**:
|
||||
- The WAL on-disk header **already carries** `wal_entry_hdr_t.term` at offset 0 of a 40-byte layout (`src/include/wal_types.h:29-39`). R1.7 ("preserve on-disk layout, value fix only") is satisfied by the format as-is. The defect is purely that `persist_t.term` (`src/include/persist.h:160`) is **never assigned** anywhere — every `hdr.term = p->term` site (`persist.c:588, 726, 1040`) therefore stamps `0`. The fix is a `persist_set_term` seam fed by the library's term, plus reading `hdr.term` back on replay (`replay_wal` reads but discards it today, `persist.c:~1500-1620`).
|
||||
- The controller uses the **new library** `src/lib/raft/raft.c` exclusively (no legacy `src/controller/raft.c` exists); ten election callbacks are already wired via `s_real_cbs` (`ha.c:1309-1320`). The log/commit callbacks defined by the three upstream specs (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) are **not yet wired** — that wiring is this spec's core deliverable.
|
||||
- **G5 (term not adopted on the replication/heartbeat path) is confirmed in code**: only `raft_on_vote_request`/`reply` route a higher term into raft state (`raft.c:125-134` → `raft_cb_persist` → `ha_persist_term`, `ha.c:1251-1294, 2441-2474`). The heartbeat consume path (`ha_consume_heartbeat_committed`, `ha.c:5013-5036`) only *drops* stale heartbeats; it never adopts a higher term. With the library now owning `UpdateTerm`-on-any-RPC, the controller must route incoming heartbeat/replication RPCs through `raft_on_append_entries_request`.
|
||||
- **G3 (reconcile is committed-boundary, not `(term,lsn)`) is confirmed**: `ha_rejoin_reconcile` (`ha_convergence.c:295-366`) clamps the truncation cutoff to `committed_lsn` and truncates the suffix blindly via `persist_truncate_wal_above` (LSN-only, `persist.c:2455-2565`). It cannot detect same-LSN/different-term divergence. The library's `raft_on_append_entries_request` + `on_truncate_from` replace this decision.
|
||||
- **R8 SNAPSHOT FEASIBILITY → RESOLVED BY DESCOPE (2026-06-21)**: the snapshot-feasibility investigation found that snapshots are a **16-byte LSN sentinel stub** today (`"XCESNAP\0"` + checkpoint_lsn; `write_snapshot` `persist.c:1068-1101`, `load_snapshot` `persist.c:1280-1307`) carrying **zero entity state**; the COW machinery a full upgrade snapshot needs (`xzarray_freeze`/`xzarray_thaw`, `zarray.c:695-757`) is built and unit-tested but **dormant** (`kvmgr.c:297-298` `vfreeze = NULL /* wired in P0-2 */`, zero production callers); the roadmap's **"entity-bearing snapshots (Task 11)"** (`roadmap.md:43`) is unstarted. Because the full configuration-upgrade materialized-state snapshot is a **HARD dependency on entity-bearing-snapshots that does not exist**, the former Requirement 8 was **descoped into the independent `config-upgrade-snapshot` spec** (which HARD-depends on `entity-bearing-snapshots`). `ha-term-safety` ships R1–R7 with **no snapshot dependency**: R7's config **marker** (consensus) is in-log, and the config **bytes** are delivered via the existing **out-of-band config-push transport** (`OPCode_replConfigPush`), hash-verified against the committed marker before activation. The config marker never depended on entity-bearing snapshots; only the descoped upgrade snapshot did.
|
||||
|
||||
## Research Log
|
||||
|
||||
### Upstream library callback contract (the seam this spec wires to)
|
||||
|
||||
- **Context**: This spec must delegate the Raft protocol to the library and wire the controller as a thin adapter, reusing the upstream callback names verbatim.
|
||||
- **Sources Consulted**: `.kiro/specs/raft-log-replication/design.md`, `.kiro/specs/raft-commit-safety/design.md`, `.kiro/specs/raft-sim-conformance/design.md`, `TLA-gap-analysis.md`.
|
||||
- **Findings**:
|
||||
- `raft-log-replication` owns: `raft_log_append`, `raft_send_append_entries`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`, `raft_append_result_t`; callbacks `on_persist_entry(ctx,index,term,entry,len)`, `on_get_entry_term(ctx,index)->term|0`, `on_send_append_entries(ctx,peer,prevIdx,prevTerm,leaderCommit,...)`, `on_truncate_from(ctx,index)`; per-peer `next_index[]`/`match_index[]`; **index 1-based == WAL LSN**; **term 0 == absent**; `UpdateTerm` applied on every RPC (G5); conflict branch = truncate-then-append in one handler call (G3).
|
||||
- `raft-commit-safety` owns: `raft_advance_commit(s)` with the Figure-8 current-term rule (G4); `on_commit_advanced(ctx,commit_index)` fired only on strict advance — **triggers** the existing barrier, never replaces it; vote/PreVote freshness becomes `(lastLogTerm,lastLogIndex)` lexicographic (G6), with the candidate's `last_log_term` added to `raft_on_vote_request`/`raft_on_pre_vote_request` signatures.
|
||||
- `raft-sim-conformance` owns the in-process oracle (Election Safety / Log Matching / Leader Completeness / State Machine Safety) — the verification cross-reference; the live regression here (R6) **complements** it (the sim proves the library; the live test proves the controller adapter wiring).
|
||||
- **Implications**: Every R1–R6 protocol invariant is realized by the library; ha-term-safety implements the **adapter callback bodies** + the term plumbing on the wire + the reconcile delegation + the live regression. The controller must NOT re-implement `logOk`, conflict truncation, the current-term commit rule, or freshness — those live in `raft.c`.
|
||||
|
||||
### WAL term stamping & persistence seam (R1, R2)
|
||||
|
||||
- **Sources Consulted**: `wal_types.h:29-59`, `persist.h:160`, `persist.c:588/726/1040/1500-1620/2455-2565`, `typemap.h:338-387`, `repl.c:239-311`.
|
||||
- **Findings**:
|
||||
- `wal_entry_hdr_t` is 40 bytes, `_Static_assert`-locked, `term` already present at offset 0 → **R1.7 is free**.
|
||||
- `persist_t.term` exists but `grep "p->term ="` / `"->term ="` returns **no assignment** in persist.c → every entry stamps term 0 (G1). Fix: a `persist_set_term(p, term)` seam called from the adapter when the library establishes/adopts a term.
|
||||
- Replication transport: the active wire format is `wal_catchup_entry_t` (`typemap.h:338-351`) which has **no term field**; the term currently rides implicitly inside the serialized header bytes. `capnp_repl_envelope_t.term` exists (`typemap.h:380-387`) but is a test-only mapped type. The adapter's `on_send_append_entries` must carry `prevLogTerm` + the entry's term explicitly so a standby persists the leader's term, not its local `p->term` (closing the `persist_wal_write_replicated` "best-effort p->term" hazard at `persist.c:719-726`).
|
||||
- `replay_wal` reads `hdr.term` but discards it; recovery must surface per-entry term to feed `on_get_entry_term` and restore `initial_log_len`/last-log-term for the library at init.
|
||||
- Next free WAL type id: `WAL_NOOP=40`, `WAL_CHECKPOINT=100` → `WAL_CONFIG_MARKER = 50` slots cleanly (R7).
|
||||
|
||||
### Term adoption, reconcile, commit, promotion seams (R2, R3, R4, R5)
|
||||
|
||||
- **Sources Consulted**: `ha.c:113-1320, 480-509, 880-1033, 5013-5036, 5338-5402`, `ha_convergence.c:240-384`, `commit.c:79-395`, `cluster.c:1200-1299`, `myapp.c` standby branch, `persist.c:691-706`.
|
||||
- **Findings**:
|
||||
- **R2/G5**: heartbeat/replication never adopt a higher term (`ha_consume_heartbeat_committed` only drops stale). Routing incoming heartbeat/repl through `raft_on_append_entries_request` makes the library do `UpdateTerm`; the adapter's `on_persist` (already `ha_persist_term`) durably records it. Monotonic-term durability before appending as leader (R2.1/2.2) is realized by the library's `BecomeLeader` + the own-term `WAL_NOOP` gate already present (`raft_cb_become_leader` arms `commit_set_writable_lsn`, `ha.c:880-1033` / `commit.c:383-395`).
|
||||
- **R3/G3**: `ha_rejoin_reconcile` committed-boundary clamp replaced by routing the leader's AppendEntries through `raft_on_append_entries_request`; the library decides match/truncate/append and calls `on_truncate_from` (adapter → `persist_truncate_wal_above`) + `on_persist_entry`.
|
||||
- **R4/G4**: `ha_compute_safe_committed_lsn` (`ha.c:480-509`) is term-blind (collects voter LSNs/tags, no term). It is superseded by the library's `commit_index` (current-term-gated) delivered via `on_commit_advanced`, which **triggers** the existing synchronous barrier. The barrier and durability (`majority_candidate_locked` self-caps at `persist_durable_lsn`, `commit.c:79-138`; persist-before-ACK; `wait_done_locked` own-term gate) are PRESERVED AS POLICY (ha-commit-durability) — only the *input* (safe commit position) changes source.
|
||||
- **R5**: clean-failover behaviors to preserve — recovery-on-promotion + truncate-to-safe (`raft_cb_become_leader`, `ha.c:917-919`), pre-promotion eligibility fence (`ha.c:933-951`, bugs 848/849), advertise-real durable floor for freshness (`raft_cb_vote_eligible`, `ha.c:299-360`), and the contiguity guard refusing a forward gap (`persist_wal_write_replicated`, `persist.c:691-706`, bug-850). These remain; the library's `(term,index)` matching is strictly additive on top.
|
||||
|
||||
### Config replication seam (R7)
|
||||
|
||||
- **Sources Consulted**: `config_repl.c:941-979, 1001-1163, 1857-1944`, `ha.c:5338-5402, 4980-5036`, `ha_convergence.c:372-384`.
|
||||
- **Findings**: config commits today through a **separate** `committed_config_tag` index (`push_config_to_standbys` waits majority ACK then `atomic_store committed_config_tag`, `config_repl.c:965-970`), carried on heartbeats (`ha.c:5343-5345`), with **no term** and tag-based anti-entropy — entirely outside the WAL `(term,lsn)` log. This is the second parallel violation. R7 replaces it: a `WAL_CONFIG_MARKER` entry (version_tag + sha256, **not bytes**) appended via `raft_log_append`, ordered/committed/reconciled in the same `(term,lsn)` log; committed config derived from the latest committed marker within the committed prefix (replaces the separate index). Config **bytes** continue to travel via the **existing out-of-band config-push transport** (`OPCode_replConfigPush`), now keyed to the marker's `{version_tag, sha256}` and hash-verified against the committed marker before activation — never in the WAL record. (The reverted byte-delivery decision: the prior draft routed bytes through a full upgrade snapshot; that snapshot is descoped to `config-upgrade-snapshot`, so the byte transport reverts to the existing config-push path.)
|
||||
|
||||
### Live regression infrastructure (R6)
|
||||
|
||||
- **Sources Consulted**: `src/controller/testsuite/waldump.{c,h}`, `test.c:129-1760`, `.kiro/specs/ha-commit-durability/repro-851/`, `INVESTIGATION-failover-job-loss.md`, `test_commit_barrier_rollout.c`, `src/tools/deploy-ha.sh`, `test-commit-durability.sh`, `roadmap.md:55`.
|
||||
- **Findings**: Infrastructure is ~90% present.
|
||||
- `waldump_path()` (`waldump.h:42`) already decodes per-entry `(LSN, TERM, TYPE, PLEN, PCRC)` and validates the CRC chain, with a payload-CRC fingerprint for cross-member comparison and nonzero exit on gaps/crc-break.
|
||||
- `xce_test` script primitives exist with exit-code verdicts: `jobsub` (captures `$jobid`), `wait-primary`, `expect-jobs N`, `expect-job $jobid`, `ctrlstatus`, `waldump`, `!shell` escape, `$?`/`$jobid` vars, `g_session_failed` → process exit code (`test.c:129-1760`).
|
||||
- `/xce-deploy-ha` + `deploy-ha.sh`/`start.sh` launch a 3-node cluster; `kill_node`/`start_node` + deterministic `--wal-sync-ms 30000` drive chaotic kills (`test-commit-durability.sh:113-198`).
|
||||
- **Gaps to build**: (a) `waldump` currently prints `TERM=0` for all entries because of G1 — once R1 lands it will show real terms; the R6 test must assert per-LSN term identity across members; (b) a `walcompare <dirA> <dirB>` cross-member `(term,lsn,pcrc)` divergence diff (already on the roadmap as planned tooling, `roadmap.md:55`) is the missing assertion primitive. R6 builds the chaotic-multi-kill script + `walcompare`, reusing all existing primitives.
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
| Option | Description | Strengths | Risks / Limitations | Notes |
|
||||
|--------|-------------|-----------|---------------------|-------|
|
||||
| **Thin callback adapter (chosen)** | Controller implements the upstream callback ports over WAL/repl/commit; library owns all protocol decisions | Honors the TLA-gap-analysis DECISION; one consistency domain; no protocol logic duplicated in controller; sim already proves the library | Requires deleting/neutralizing the term-blind controller logic (`ha_compute_safe_committed_lsn`, committed-boundary reconcile) rather than extending it | Matches Ports-and-Adapters already established by the election callbacks |
|
||||
| Patch the controller's split log (rejected) | Add `(term,lsn)` matching directly into `ha_convergence.c` + `ha_compute_safe_committed_lsn` | No library dependency | Re-implements raft.tla in the controller; two consistency domains; cannot be sim-verified; explicitly rejected by the 2026-06-21 DECISION | Rejected |
|
||||
| Config bytes in WAL (rejected) | Carry config bytes in `WAL_CONFIG_MARKER` payload | Self-contained log | Bloats the log; R7.2/R7.4 forbid it | Rejected by requirements |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Decision: Unify R1–R6 protocol onto the library; controller is a thin adapter
|
||||
|
||||
- **Context**: TLA-gap-analysis DECISION (2026-06-21) mandates the library own the protocol; the controller adapts.
|
||||
- **Alternatives Considered**: (1) thin adapter (chosen); (2) patch controller split-log (rejected — duplicates protocol, unverifiable).
|
||||
- **Selected Approach**: Wire five new library callbacks to controller I/O sites; route incoming heartbeat/replication through `raft_on_append_entries_request`; drive append via `raft_log_append`; replace the term-blind safe-commit and committed-boundary reconcile with the library's `commit_index`/`on_truncate_from`. No protocol logic is added to the controller.
|
||||
- **Rationale**: Single consistency domain; the deterministic sim already verifies the protocol, so the live regression need only prove the *wiring*.
|
||||
- **Trade-offs**: Must neutralize existing term-blind controller paths (a deletion, not an extension); upstream signature changes are revalidation triggers.
|
||||
- **Follow-up**: Confirm the upstream callback/entry-point signatures are stable before wiring (Revalidation Triggers below).
|
||||
|
||||
### Decision: Generalize job-state and config-change onto one `(term,lsn)` replicated log (R7)
|
||||
|
||||
- **Context**: Config is a second parallel consistency domain (`committed_config_tag`, no term) exposed to the same divergence.
|
||||
- **Selected Approach**: Record each config change as a lightweight `WAL_CONFIG_MARKER` (version_tag + sha256, no bytes) appended via `raft_log_append` so it inherits term-safety, reconcile, and commit. Derive committed config from the latest committed marker; retire the separate `committed_config_tag` commit index.
|
||||
- **Rationale**: One log, one reconcile, one commit rule — the Generalization lens: config change is a special case of "an ordered committed log entry".
|
||||
- **Trade-offs**: Config bytes are delivered out-of-band via the existing config-push transport (`OPCode_replConfigPush`), hash-verified against the committed marker before activation — no snapshot dependency. During migration, `committed_config_tag` becomes a derived value, not an authoritative index.
|
||||
- **Follow-up**: R7's marker is implementable now (no entity-bearing snapshots needed); the config bytes ride the existing config-push install path. The full materialized-state upgrade snapshot is descoped to `config-upgrade-snapshot` — see the resolved decision below.
|
||||
|
||||
### Decision (RESOLVED 2026-06-21): former R8 descoped into `config-upgrade-snapshot`; R7 reverts to out-of-band config-push byte delivery
|
||||
|
||||
- **Context**: The former Requirement 8 required a **full entity-bearing state snapshot** embedding the new config, installed on every member, so no member replays pre-upgrade WAL under the new config (the bug-857 hazard).
|
||||
- **Finding (the prior blocker)**: Snapshots are a 16-byte LSN sentinel (`persist.c:1068-1101/1280-1307`); the entity-serialization machinery (`xzarray_freeze/thaw`) is dormant (`kvmgr.c:297`); roadmap "entity-bearing snapshots (Task 11)" is **unstarted** (`roadmap.md:43`). The full upgrade snapshot is therefore a HARD dependency on entity-bearing-snapshots, which does not exist.
|
||||
- **Alternatives Considered**:
|
||||
1. Make entity-bearing-snapshots a hard prerequisite and sequence the upgrade snapshot after it, inside this spec.
|
||||
2. Carry config bytes in the WAL marker — **violates R7.2/R7.4**.
|
||||
3. Per-node replay under new config — the exact bug-857 hazard.
|
||||
4. **Descope the materialized-state-on-upgrade work into an independent spec (selected)** — split the entity-snapshot blocker out of `ha-term-safety` so R1–R7 can ship now.
|
||||
- **Selected Approach**: The configuration-upgrade materialized-state consistency (former R8) is **descoped into the independent `config-upgrade-snapshot` spec**, which **HARD-depends on `entity-bearing-snapshots`**. `ha-term-safety` retains R7's consensus half (the in-log `(term,lsn)` config marker) and **reverts** the configuration-byte delivery to the **existing out-of-band config-push transport** (`OPCode_replConfigPush`), keyed to the marker's `{version_tag, sha256}` and hash-verified against the committed marker before activation. No snapshot is built or depended on here. `config-upgrade-snapshot` later supersedes this interim byte delivery for the upgrade case once entity-bearing snapshots land.
|
||||
- **Rationale**: Do not paper over an infeasible requirement, and do not block the implementable R1–R7 (protocol term-safety + config marker) on an unstarted entity-serialization backlog. The marker + commit + reconcile (R7) never required entities; only the materialized-state install did. The byte delivery falls back to the already-working config-push path.
|
||||
- **Trade-offs**: `ha-term-safety` no longer delivers the full upgrade-snapshot guarantee (now owned by `config-upgrade-snapshot`); the interim config-push byte delivery is superseded for the upgrade case once that spec lands.
|
||||
- **Follow-up**: `config-upgrade-snapshot` brief + requirements written (`../config-upgrade-snapshot/{brief.md,requirements.md}`); it anchors on this spec's `(term,lsn)` config marker and consumes `entity-bearing-snapshots`.
|
||||
|
||||
### Decision: Preserve the commit barrier + durability + clean-failover behaviors as policy
|
||||
|
||||
- **Context**: R4 and R5 must not regress ha-commit-durability (bugs 661/828) or bugs 848/849/850.
|
||||
- **Selected Approach**: `on_commit_advanced` only **triggers** the existing synchronous barrier; `majority_candidate_locked` self-cap, persist-before-ACK, the pre-promotion fence, advertise-real-durable-floor freshness, and the contiguity guard are unchanged. The library supplies the safe commit position; the durable act stays in the controller.
|
||||
- **Rationale**: The barrier/durability is verified by tests + the Protocol Verification Checklist (not by raft.tla, which assumes a durable log); changing source-of-truth for the commit position must not change the durability mechanism.
|
||||
- **Trade-offs**: None functional; R4.x latency for the steady state is unchanged (R5.4) because the barrier path is untouched.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- **Former R8 infeasible without entity-bearing snapshots** → RESOLVED by descoping the configuration-upgrade materialized-state work into the independent `config-upgrade-snapshot` spec (HARD-depends on `entity-bearing-snapshots`); `ha-term-safety` ships R1–R7 with out-of-band config-push byte delivery and no snapshot dependency.
|
||||
- **Upstream signature drift** (the three library specs are generated but not yet implemented) → Listed as Revalidation Triggers; the adapter wires to the exact published signatures; if a signature changes, re-check the adapter table.
|
||||
- **Neutralizing term-blind controller paths could regress clean failover** → R5 acceptance criteria + the existing durability tests are the guard; the live R6 regression asserts both chaotic AND clean convergence.
|
||||
- **`waldump` shows TERM=0 pre-fix** → expected (G1); R6 asserts non-zero, per-LSN-consistent terms post-fix, and the negative direction (pre-fix repro must fail) per R6.5.
|
||||
- **Migration of `committed_config_tag` to marker-derived** → derive-not-store; on first boot after upgrade, a cluster with no marker treats config as term-0/lowest-known (R1.8 analogue) until the first marker commits.
|
||||
|
||||
## References
|
||||
|
||||
- `.kiro/specs/ha-term-safety/requirements.md` — R1–R7 (former R8 descoped 2026-06-21).
|
||||
- `.kiro/specs/config-upgrade-snapshot/{brief.md,requirements.md}` — the descoped configuration-upgrade materialized-state snapshot (HARD-depends on `entity-bearing-snapshots`).
|
||||
- `.kiro/specs/ha-term-safety/TLA-gap-analysis.md` — gaps G1–G6 + the library-owns-protocol DECISION.
|
||||
- `.kiro/specs/raft-log-replication/design.md`, `.kiro/specs/raft-commit-safety/design.md`, `.kiro/specs/raft-sim-conformance/design.md` — upstream callback contracts (consumed verbatim).
|
||||
- `.kiro/steering/persistence.md`, `ha-async-io.md`, `roadmap.md:13-21,43,55` — durability policy, process-lifecycle invariant, Raft conformance series, Task 11 backlog, planned `walcompare` tooling.
|
||||
- `.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md` + `repro-851/` — bug-851/857 evidence and the regression baseline.
|
||||
22
.kiro/specs/ha-term-safety/spec.json
Normal file
22
.kiro/specs/ha-term-safety/spec.json
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"feature_name": "ha-term-safety",
|
||||
"created_at": "2026-06-21T09:34:18Z",
|
||||
"updated_at": "2026-06-21T12:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "design-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false
|
||||
}
|
||||
106
.kiro/specs/ha-wal-replication/research.md
Normal file
106
.kiro/specs/ha-wal-replication/research.md
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
# Research: HA WAL Replication
|
||||
|
||||
This document captures the codebase-grounded research and gap analysis for the
|
||||
`ha-wal-replication` spec — bridging the WAL replicator (`repl.c`) and the HA
|
||||
connection layer (`ha.c`/`cluster.c`/`config_repl.c`) so WAL entries written on
|
||||
the primary are transmitted to standbys, applied + ACKed, and majority-committed
|
||||
before the caller is acknowledged.
|
||||
|
||||
There was no prior `research.md`; this file is created fresh.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
### STATUS: PARTIAL (code-complete; only the live 3-node integration test is open)
|
||||
|
||||
Every code path required by the 5 requirement groups is implemented and unit-
|
||||
tested in the controller suite under ASAN. The single open item that is genuine
|
||||
work is **5.4** — the live 3-node `/xce-deploy-ha` integration test — which is an
|
||||
environment-dependent manual test, not a code gap. The remaining 5 open `[ ]`
|
||||
checkboxes are all **group-header artifacts** (the parent bullets `1`, `2`, `3`,
|
||||
`4`, `5`), whose child tasks are all `[x]`.
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
Tag legend: **MET** (verified in code), **Constraint** (design boundary, behaves
|
||||
as specified), **Unknown** (only verifiable on a live cluster), **Missing**.
|
||||
|
||||
| Req | Statement (abbrev.) | Tag | Evidence (file:line) |
|
||||
|-----|---------------------|-----|----------------------|
|
||||
| 1.1 | Register standby with replicator on connect (election victory + late-joiner) | **MET** | `ha.c:1060-1074` (become-leader promote loop calls `repl_add_standby`); `ha.c:5035-5049` (late-joiner `on_standby_connected` path) |
|
||||
| 1.2 | Deregister on disconnect (RDHUP / error) | **MET** | `ha.c:3961-3964` (`vote_ep_err_handler` → `repl_remove_standby`); `ha.c:5089-5091` (`standby_ep_err_handler_primary`); `ha.c:5258` (evicted batch) |
|
||||
| 1.3 | Idempotent re-registration | **MET** | `repl_add_standby` matches existing slot by ep pointer; covered by `test_p2_integration.c:220-235` (`test_repl_api_completeness`) and registration-lifecycle assertions |
|
||||
| 2.1 | `repl_broadcast` serializes + enqueues per non-catch-up standby via `xce_request` | **MET** | `repl.c:279-307` (builds `wal_catchup_entry_t`, calls `xce_wal_batch_req_raw`); `test_repl_unit.c:222-231` (send-hook captures payload round-trip) |
|
||||
| 2.2 | Skip standby while `in_catch_up` set | **MET** | `repl.c:265-266` (`atomic_load in_catch_up` → continue) |
|
||||
| 2.3 | Pending-ACK over `MAX_PENDING_ACKS` → trigger catch-up + skip | **MET** | `repl.c:268-277` (inline start-catchup + continue); `test_repl_majority.c:282` (over-limit eviction) |
|
||||
| 2.4 | No standbys → return 0 immediately | **MET** | `repl.c:251-257` (early return, WARN) |
|
||||
| 3.1 | Standby applies entry to persistent storage + advances `applied_lsn` | **MET** | `ha.c:1558-1588` (`ha_apply_single_entry` → `persist_wal_write_replicated` then `apply_wal_entry`); `ha.c:1676` `ha_advance_applied_lsn` |
|
||||
| 3.2 | After apply, send ACK carrying applied LSN | **MET** | `ha.c:1591-1613` (`ha_send_wal_ack_reply`, `OPCode_walAck` + `walackreplymsg`); called at `ha.c:1650`, `ha.c:1679` |
|
||||
| 3.3 | Apply failure → log, skip, no ACK | **MET** | `ha.c:1573-1579`, `1640-1647`, `1666-1672` (return non-zero, no ACK) |
|
||||
| 3.4 | Standby stays in `CTRL_STANDBY`; dispatch gate admits walCatchupEntry | **MET** | `cluster.c:163-187` (`handle_wal_catchup_entry`, gate already admits); `ha_dispatch_gate` admits `walCatchupEntry` in STANDBY/ELECTING/INIT |
|
||||
| 4.1 | `repl_wait_majority_ack` counts standbys ≥ target, quorum incl. self-ack | **MET** | `repl.c:422+` (`repl_wait_majority_ack`); `test_repl_majority.c`, `test_commit_engine.c` quorum tests |
|
||||
| 4.2 | Timeout before quorum → -ETIMEDOUT; do not advance `committed_lsn` | **MET** | `persist.c:1898-1919` (`sync_err = repl_wait_majority_ack(...)`, only advances on success); `test_commit_engine.c:922-949` (`test_standby_removal_below_majority_wait_fails_at_deadline`, asserts -ETIMEDOUT + no regress) |
|
||||
| 4.3 | On WAL ACK → update `acked_lsn` + signal ack cond | **MET** | `ha.c:3520-3531` (`repl_ack_decode_and_dispatch`: `read_WalAckReply` → `repl_on_ack`); `repl_on_ack` updates `acked_lsn` + signals `ack_cond` (`repl.c:194+`) |
|
||||
| 4.4 | Single-node (no standbys) → return 0 + advance to written LSN | **MET** | `repl_wait_majority_ack` self-ack path; `test_p2_integration.c:238+` (single-node production-contract test) |
|
||||
| 5.1 | Apply entries in LSN order; out-of-order buffered/rejected | **MET (Constraint)** | Batch loop applies in delivered order (`ha.c:1631-1654`); `persist_wal_write_replicated` is idempotent on re-stream and uses `e->lsn` directly (no reorder). Contiguous-prefix commit ordering enforced upstream by the single append lock (see `src/controller/CLAUDE.md` Commit Ordering). |
|
||||
| 5.2 | DEBUG3/TRACE log opcode + target LSN on transmit | **MET** | `repl.c:304-305` (`TRACE("repl_broadcast: sent lsn=%lu type=%u to %s:%u")`) |
|
||||
| 5.3 | INFO log opcode + quorum count when quorum reached | **MET** | `repl_wait_majority_ack` quorum-reached logging (repl.c); 5.x tests assert quorum path |
|
||||
| 5.4 | WARN log on timeout w/ opcode, timeout, per-standby ACK state | **MET** | `repl.c` timeout WARN path; verified by majority-timeout unit test |
|
||||
| 5.5 | All new paths pass controller suite under ASAN, zero new leaks | **MET** | tasks.md 5.5 note: 45 tests / 308 assertions pass; ASAN leaks pre-existing in unrelated tests only |
|
||||
|
||||
**Score: 22/22 EARS requirements MET** (4.x/2.x/3.x/5.x all verified in code +
|
||||
unit tests). The only requirement whose *end-to-end* behavior is not yet proven
|
||||
on real hardware is the failover-durability slice of 4.1/5.5 exercised by task
|
||||
5.4 — that is an Unknown pending the live cluster, not a code Missing.
|
||||
|
||||
### Open Tasks — Classification
|
||||
|
||||
| Task | Text | Classification | Notes |
|
||||
|------|------|----------------|-------|
|
||||
| `1` | repl_broadcast TCP send (parent) | **group-header-artifact** | children 1.1, 1.2 both `[x]`; `repl.c:239-311` fully implemented |
|
||||
| `2` | HA→replicator registration bridge (parent) | **group-header-artifact** | children 2.1–2.3 `[x]`; bridge wired at `ha.c:1060`, `5043`, `3964`, `5091` |
|
||||
| `3` | Standby WAL ACK reply (parent) | **group-header-artifact** | children 3.1–3.3 `[x]`; `ha_send_wal_ack_reply` + `handle_wal_catchup_entry` live |
|
||||
| `4` | Primary ACK handler → advance committed_lsn (parent) | **group-header-artifact** | children 4.1, 4.2 `[x]`; `read_WalAckReply`→`repl_on_ack` at `ha.c:3520` |
|
||||
| `5` | Validation: tests + integration (parent) | **group-header-artifact** | children 5.1, 5.2, 5.3, 5.5 `[x]`; only 5.4 remains |
|
||||
| `5.4` | Integration: 3-node cluster WAL replication | **env-dependent-test** (the one genuine remaining item) | Requires live `/xce-deploy-ha` cluster; explicitly marked `_Blocked_` / not automatable in the unit suite |
|
||||
|
||||
**Doc-drift note (non-blocking):** `design.md` predicted the unit tests would
|
||||
live in a new file `src/controller/testsuite/test_ha_wal_repl.c`. That file was
|
||||
never created; the 5.1/5.2/5.3 unit tests actually landed in existing files —
|
||||
`test_repl_unit.c` (broadcast send-hook + payload round-trip, lines 178-487) and
|
||||
`test_repl_majority.c`/`test_p2_integration.c` (ACK round-trip, registration
|
||||
lifecycle, over-limit eviction). Coverage is present; only the filename in the
|
||||
design is stale. No action required unless the team wants the design corrected.
|
||||
|
||||
### Genuine Remaining Work — Options
|
||||
|
||||
The only genuine open item is **task 5.4** (live 3-node integration test). It is
|
||||
a verification gap, not a code gap.
|
||||
|
||||
**Prerequisite for 5.4:** a running multi-node cluster via `/xce-deploy-ha`
|
||||
(e.g. `/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993`). Cannot run
|
||||
inside the unit/ASAN suite — needs three live controller processes, a primary
|
||||
election, `jobsub`/`joblist`, and a primary SIGKILL to exercise failover
|
||||
durability. (`ha-commit-durability` already shipped a related rolling-SIGKILL
|
||||
soak that proves zero acked-job loss; 5.4 is the WAL-replication-specific
|
||||
visibility + commit-advance assertion on top of that.)
|
||||
|
||||
| Option | Description | Effort | Risk |
|
||||
|--------|-------------|--------|------|
|
||||
| **A — Run 5.4 as the manual live test (as specified)** | Deploy 3 nodes via `/xce-deploy-ha`, submit a job, assert it appears in `joblist` on primary + all standbys, kill primary, assert promoted standby has the job and `committed_lsn` advanced on all nodes. Mark 5.4 `[x]` with captured evidence. | **Low** (~1–2 h, mostly orchestration + observation) | **Low** — code paths already unit-verified and overlap the proven commit-durability soak; main risk is environment flakiness (port binding, election timing) not correctness |
|
||||
| **B — Automate 5.4 as a scripted multi-process harness** | Wrap the deploy/submit/kill/verify flow in a repeatable script (CI-runnable) so the integration test is regression-protected, not one-shot manual. | **Medium** (~0.5–1 day) | **Medium** — multi-process orchestration in CI is flaky-prone (timing, cleanup); higher value long-term but exceeds the spec's stated "manual integration test" boundary |
|
||||
| **C — Defer / close on the commit-durability soak as proxy** | Treat the already-passing `ha-commit-durability` rolling-failover soak (zero acked-job loss, cluster stays available) as sufficient end-to-end evidence and close 5.4 by reference, without a dedicated WAL-replication run. | **Minimal** | **Medium** — the durability soak proves *durability* but not the *standby-visibility* assertion (job appears in `joblist` on standbys) that 5.4 specifically calls out; closing without that leaves a small explicit coverage hole |
|
||||
|
||||
**Recommendation:** Option A — run 5.4 manually as specified once a cluster is
|
||||
available; it is low-effort, low-risk, and directly discharges the only open
|
||||
requirement-evidence gap. Consider Option B later only if WAL replication
|
||||
regressions become a recurring concern.
|
||||
|
||||
### Bottom line
|
||||
|
||||
`ha-wal-replication` is **functionally code-complete**: all 22 EARS requirements
|
||||
are implemented and unit-tested under ASAN. The 5 open parent checkboxes are
|
||||
group-header artifacts. The single substantive open item is the live 3-node
|
||||
integration test (5.4), which is environment-gated on `/xce-deploy-ha` and best
|
||||
discharged as the specified manual run (Option A).
|
||||
|
|
@ -125,3 +125,124 @@ endpoint/ctx state instead of leaving it dead.
|
|||
- **R-redirect-lifetime:** `redirect_host` is a decoded capnp string freed with the reply; must be
|
||||
copied into a ctx-owned buffer (D3) to avoid use-after-free (cf. the `init_reply_by_req`
|
||||
memset hazard already documented at request.c:243-250).
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
**Method:** Forward-looking requirements-vs-codebase verification. Each requirement traced to a
|
||||
real symbol/file:line in `src/lib`, `src/include`, `src/controller`. Every `[x]` task confirmed
|
||||
against actual code (not just the checkbox). The only genuinely-open leaf task (6.2) re-scoped for
|
||||
the remaining design/exec work.
|
||||
|
||||
### Verdict
|
||||
|
||||
**STATUS: PARTIAL — code-complete, live-E2E-incomplete.** All implementation tasks (1.1–6.1 + 5.3)
|
||||
are present and wired in real code; the sole open leaf is **6.2 (live 3-node failover E2E)**, which
|
||||
Implementation Notes flag as *partial* (server-side failover proven + 2 live bugs found/fixed, but
|
||||
clean within-session client transparent-recovery never visually demonstrated in the sandbox env).
|
||||
The 6 unchecked `[ ]` items besides 6.2 are **phase-group headers** (tasks 1/2/3/4/5/6), not work
|
||||
units — all their leaf sub-tasks are `[x]` and verified below.
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
| Req | Summary | Tag | Evidence (file:line) |
|
||||
|-----|---------|-----|----------------------|
|
||||
| 1.1 | All ops (read+write) through failover | **MET** | `client.c` — 21× `XCECLI_FAILOVER_CALL`/`XCECLI_FF_CALL`; macro in `client_internal.h:152` |
|
||||
| 1.2 | Next request completes on new primary | **MET** (unit) / **Unknown** (live) | `client_failover.c:xcecli_retry_next` + `xce_ctx_reconnect` (`xcectx.c:320`); live not visually confirmed (6.2) |
|
||||
| 1.3 | Public API signatures unchanged | **MET** | Notes 5.1 "zero signature changes"; macro wraps op bodies, no proto edits |
|
||||
| 2.1 | Unreachable → try other masters | **MET** | `xce_failover_next_target` (cursor advance+wrap over `conf->masters[]`), `client_failover.c` |
|
||||
| 2.2 | Redirect → rotate, hint-first | **MET** | hint-first/consume then cursor; `xce_failover_rotate` |
|
||||
| 2.3 | Standby → treat as redirect/rotate | **MET** | classifier `client_failover.c:59` (`E_XCE_STANDBY`→retry-now) |
|
||||
| 2.4 | Sticky current primary | **MET** | `cur_master` in `xce_failover_state_t` (`client_internal.h`) |
|
||||
| 2.5 | Redirect/standby/no-primary ≠ success | **MET** | classifier never maps these to SUCCESS; unit `test_libxce_retry.c` |
|
||||
| 3.1 | No-primary → backoff to budget | **MET** | `client_failover.c:64` (NOMASTER/NOT_READY→backoff) |
|
||||
| 3.2 | Success before budget | **MET** (unit) | `test_libxce_retry.c` become-primary-mid-loop |
|
||||
| 3.3/3.4 | Budget exhausted → distinct cluster-not-ready (not EINVAL) | **MET** | `*rc = -E_XCE_NOT_READY` (`client_failover.c:282,317`) |
|
||||
| 4.1–4.5 | Retriable vs terminal classification | **MET** | `xce_failover_classify`, 11-row taxonomy in `failover_harness.h`; tests pass |
|
||||
| 5.1 | Reads+writes failover | **MET** | OpCategory: writes=true for jobsub/jobctrl/join/… (Notes 5.1) |
|
||||
| 5.2 | Write retry not double-applied (idempotent DUP→success) | **MET** (client) / **Constraint** | classifier DUP-on-resent-write→success; relies on controller `job.c:56,95` jobid replay guard |
|
||||
| 5.3 | No silent drop | **MET** | loop returns success or distinct error |
|
||||
| 6.1/6.2 | Configurable params + safe defaults | **MET** | `xce_retry_config_t` + `XCE_RETRY_DEFAULTS` (`request.h:183-193`, max_retries=20) |
|
||||
| 6.3 | Jitter / anti-thundering-herd | **MET** | `test_decide_jitter_staggers_many_clients` (64-carrier, Notes 6.1) |
|
||||
| 7.1/7.2 | Log rotation target + exhaustion count | **MET** | `tlog_info`/`tlog_warn` in `xcecli_retry_next` (Notes 3.1) |
|
||||
| 8.1 | Redirect carries primary host/port | **MET** | `fill_redirect_reply` (`join.c:28-31`) ← `ha.primary_host/port` (`ha.h:149-150`) |
|
||||
| 8.2 | Unknown primary → no-primary, not addressless redirect | **MET** | `join.c:289` `-E_XCE_NOMASTER` path unchanged |
|
||||
| 8.3/8.4 | Client uses hint best-effort, falls back to rotation | **MET** | `xce_ctx_capture_redirect_hint` (`xcectx.c:30,57`); rotate consumes-then-falls-through |
|
||||
|
||||
**Orphan cleanup (task 1.1):** confirmed removed — `xce_request_with_retry()` gone, only a
|
||||
tombstone comment remains (`request.c:1927`).
|
||||
**Controller audit (task 5.3):** `join.c` confirmed sole `E_XCE_REDIRECT` emitter; controller test
|
||||
`test_join_redirect.c` registered (`controller_test.c:155`, `meson.build:97`).
|
||||
|
||||
Tally: **24/24 acceptance criteria MET in code** (1.2 live-unconfirmed; 5.2 carries one external
|
||||
Constraint = controller idempotency, which is in-scope-consumed not in-scope-built).
|
||||
|
||||
### Open Tasks — Forward Gap Analysis
|
||||
|
||||
Only one leaf task is open. The remaining `[ ]` are phase headers that auto-close when their
|
||||
sub-tasks are `[x]` (all are) — recommend marking 1/2/3/4/5 complete; keep 6 open until 6.2 lands.
|
||||
|
||||
#### Task 6.2 — Live 3-node failover end-to-end *(the real PENDING work)*
|
||||
|
||||
**What is missing (vs. what is done):**
|
||||
- DONE in-env: 3-node `deploy-ha.sh` cluster elected a primary; killing it triggered server-side
|
||||
failover (ctrl-2 → term=2); the run REPRODUCED the original `Invalid argument` bug and the failover
|
||||
loop now ENGAGES (`xcecli_joblist`→`xcecli_retry_next`→`xce_failover_rotate`). Two live-only bugs
|
||||
found+fixed with regression unit tests: **bug-609** (fresh-call-after-disconnect `-EINVAL`
|
||||
classified terminal → known-dead-ctx override in `decide`) and **bug-610** (`xce_failover_rotate`
|
||||
SEGV — reconnect never re-bound `ep->iomgr`/handlers; fixed via new `xce_ctx_reconnect()`).
|
||||
- MISSING: a clean, *visually-captured* within-session demonstration of the three acceptance
|
||||
outcomes against a stable runtime: (a) `joblist` reports cluster-not-ready during the election
|
||||
window then transparently recovers with NO `EINVAL`; (b) a request to a standby lands on the
|
||||
primary in one hop via the populated redirect hint (and via rotation when hint absent); (c) a
|
||||
`jobsub` spanning the failover creates **exactly one** job (idempotency end-to-end).
|
||||
|
||||
**Integration points to exercise:** `/xce-deploy-ha localhost:9991 9992 9993` → `xce_test`
|
||||
(`set masters`, `client`, `jobsub`, `joblist`) → kill primary → re-`joblist`. NOTE the build hazard
|
||||
recorded twice in Notes: **building `lib` removes the controller binary from the shared build dir** —
|
||||
must `xce-build.sh controller` (relinks `xce_test`) before each deploy.
|
||||
|
||||
**Options:**
|
||||
- **Option A (recommended): stable-env manual E2E.** Run the reproduction in a non-sandbox env where
|
||||
backgrounded controller procs survive between steps (the sandbox reaps them). Capture the 3 outcomes.
|
||||
*Effort S–M, Risk Low* — code already proven to engage; this is confirmation + evidence capture.
|
||||
- **Option B: scripted deterministic E2E harness** (a `testsuite` shell/expect script that boots the
|
||||
cluster, kills the primary on a signal, asserts on `xce_test` output). *Effort M, Risk Med* — gives
|
||||
a repeatable regression gate but adds harness maintenance; process-lifecycle flakiness is the risk.
|
||||
- **Option C: in-process integration test** simulating the controller side with fakes (no real procs).
|
||||
*Effort M–L, Risk Med* — avoids deploy flakiness but re-tests what unit tests already cover (low
|
||||
marginal value vs. real-cluster confidence); does NOT satisfy 6.2's "original reproduction" intent.
|
||||
|
||||
**Recommendation:** Option A to *close* 6.2 (intent = the original bug reproduction passes live),
|
||||
optionally backstopped by Option B as a CI regression gate in a follow-on.
|
||||
|
||||
#### Phase headers 1–5 (and 6) — administrative
|
||||
|
||||
*Gap:* checkbox-only. *Action:* flip 1/2/3/4/5 to `[x]` (all sub-tasks verified `[x]` above); leave
|
||||
6 `[ ]` until 6.2 closes. *Effort S, Risk None.*
|
||||
|
||||
### Research-Needed (for any remaining design/exec)
|
||||
|
||||
1. **Stable runtime for live E2E** — identify an env where backgrounded `xce_controller` procs
|
||||
persist across orchestration steps (sandbox reaps them; nix runtime env needed). Blocks 6.2-A.
|
||||
2. **Build-order guard** — codify "rebuild controller after any `lib` build" (Notes 3.1/6.2 twice).
|
||||
Consider a `xce-build.sh` ordering note or a make dependency so `xce_test` is never stale-linked.
|
||||
3. **Idempotency end-to-end proof (5.2)** — confirm the controller's client-`jobid` replay guard
|
||||
(`job.c:56,95`) actually returns `E_XCE_DUP_JOB`/`-EEXIST` on the *re-sent* write across a real
|
||||
failover (unit-mocked today; live single-job-created assertion is the open evidence).
|
||||
4. **Downstream readiness (out-of-boundary, flag only)** — `component-failover-adoption` (HARD dep)
|
||||
still owes: per-daemon **full master-list** wiring (today even CLI uses `masters[0]`,
|
||||
`client.c:131`) + **liveness-safe** bounded-blocking verification for async C daemons (worker
|
||||
`controller_connector.c`, scheduler) so backoff never stalls the heartbeat/iomgr thread. Not 6.2's
|
||||
scope, but it is the gate before this failover mechanism delivers value to real daemons.
|
||||
|
||||
### Integration Risks (remaining)
|
||||
|
||||
- **Process-lifecycle flakiness** in any live/scripted E2E (controller procs, port reuse, election
|
||||
timing) — the dominant risk for 6.2; mitigated by Option A in a stable env + jittered election.
|
||||
- **Stale `xce_test` link** if controller isn't rebuilt after a `lib` build — would mask a real fix
|
||||
or surface a phantom one. Mitigation = Research-Needed #2.
|
||||
- **Idempotency dependency** (5.2) is external (controller `job.c`) — a regression there breaks
|
||||
exactly-once across failover without touching libxce. Mitigation = Research-Needed #3.
|
||||
|
||||
|
|
|
|||
31
.kiro/specs/raft-commit-safety/brief.md
Normal file
31
.kiro/specs/raft-commit-safety/brief.md
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# Brief: raft-commit-safety
|
||||
|
||||
## Problem
|
||||
After `raft-log-replication` gives the library a real `(term, index)` log, two `raft.tla` commit/election-safety rules are still missing in XCE (gap analysis: `.kiro/specs/ha-term-safety/TLA-gap-analysis.md`):
|
||||
- **G4** — commit-index advance is **term-blind**. `raft.tla` `AdvanceCommitIndex` (line 259) only advances `commitIndex` to `Max(agreeIndexes)` **if `log[Max].term == currentTerm`** (the Figure-8 safety rule: a leader never commits a previous-term entry by replica-count alone). XCE's commit barrier + `ha_compute_safe_committed_lsn` advanced committed "from 1 granted voter" with no current-term check (because the WAL term was dead); the own-term `WAL_NOOP` gate is only a partial mitigation.
|
||||
- **G6** — vote freshness is **LSN-only**. `raft.tla` `HandleRequestVoteRequest` `logOk` (line 285) compares `(lastLogTerm, lastLogIndex)` lexicographically; the library's `on_vote_eligible` gate uses `config_tag ≥` + `applied LSN ≥` only — it never compares `lastLogTerm`.
|
||||
|
||||
## Approach
|
||||
Add the commit-safety and freshness rules to the library, strictly per `raft.tla`, on top of the `raft-log-replication` log model.
|
||||
|
||||
### TLA+ actions implemented
|
||||
- `AdvanceCommitIndex(i)` (raft.tla:259) → `raft_advance_commit(...)`: compute `agreeIndexes` from per-peer `matchIndex` (a quorum agrees), and advance `commitIndex` to `Max(agreeIndexes)` **only when `on_get_entry_term(Max) == currentTerm`**. Otherwise hold.
|
||||
- `HandleRequestVoteRequest` `logOk` (raft.tla:285) → upgrade the freshness gate to `(lastLogTerm, lastLogIndex)`: grant only if `cand.lastLogTerm > local.lastLogTerm ∨ (== ∧ cand.lastLogIndex ≥ local.lastLogIndex)`, applied to both `raft_on_vote_request` and the PreVote eligibility path.
|
||||
|
||||
## Authoritative shared contract
|
||||
Uses the `raft-log-replication` contract verbatim (1-based index = WAL LSN; per-entry `uint64_t term`; `on_get_entry_term`, per-peer `matchIndex`). New callback:
|
||||
- `void on_commit_advanced(void *ctx, uint64_t commit_index)` — fired AFTER `commitIndex` advances; the controller adapter feeds this into the **existing synchronous commit barrier / durability** (persist-before-ACK preserved — this callback does not replace the barrier, it triggers commit-eval).
|
||||
Vote RPC freshness gains `last_log_term` alongside the existing `last_log_index`/`config_tag` carried to `on_vote_eligible` (extend the eligibility signature or the vote-request payload accordingly).
|
||||
|
||||
## Scope
|
||||
- **In**: `AdvanceCommitIndex` with the current-term rule; `(lastLogTerm, lastLogIndex)` vote/PreVote freshness; `on_commit_advanced` callback wired to the controller commit barrier; simulator assertion of **Leader Completeness** (a committed entry is present in every future leader's log).
|
||||
- **Out**: the log model + AppendEntries (owned by `raft-log-replication`); the conformance harness (`raft-sim-conformance`); config markers/snapshot (`ha-term-safety`); the barrier/durability mechanism itself (controller — this spec only triggers it via `on_commit_advanced`).
|
||||
|
||||
## Preserve as policy
|
||||
The synchronous commit barrier + persist-before-ACK durability is STRONGER than canonical Raft async apply and MUST remain: `on_commit_advanced` triggers the existing barrier; it does not weaken it. PreVote and priority leadership unchanged.
|
||||
|
||||
## Verification
|
||||
Extend `raft_sim.c`: construct Figure-8 scenarios (a previous-term entry replicated to a quorum but not yet committed by the current leader) and assert the library refuses to commit it until a current-term entry is committed; assert **Leader Completeness** across forced leader changes. Replace `ha_compute_safe_committed_lsn`'s term-blind advance with the library's current-term-gated `commitIndex`.
|
||||
|
||||
## Dependencies
|
||||
`raft-log-replication` (HARD — needs the `(term,index)` log + `matchIndex`). Feeds `raft-sim-conformance` and `ha-term-safety`. Closes bug-851 G4/G6.
|
||||
392
.kiro/specs/raft-commit-safety/design.md
Normal file
392
.kiro/specs/raft-commit-safety/design.md
Normal file
|
|
@ -0,0 +1,392 @@
|
|||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose**: This feature adds the **commit-safety and election-freshness** rules of `raft/raft.tla` (Ongaro canonical) to the pure Raft library (`src/lib/raft/raft.{c,h}`), on top of the log half delivered by `raft-log-replication`. It closes the two remaining safety gaps of bug-851: **G4** (term-blind commit advance) and **G6** (LSN-only vote freshness).
|
||||
|
||||
**Users**: The XCE controller (as a Raft node, which will route its commit barrier through the new entry point in the downstream `ha-term-safety` integration), the deterministic simulator (`raft_sim.c`), and the downstream `raft-sim-conformance` spec (Leader Completeness oracle).
|
||||
|
||||
**Impact**: Today the commit position advances **term-blind** in the controller (`ha_compute_safe_committed_lsn` + the commit barrier) — it can mark a previous-term entry committed by replica-count alone (G4) — and vote freshness compares LSNs only, never the last-log term (G6). After this feature, the library owns `commit_index` and advances it exactly as `raft.tla` `AdvanceCommitIndex` does (quorum agreement **and** current-term entry), notifying the controller via `on_commit_advanced` to **trigger** the existing synchronous commit barrier (the barrier itself is unchanged and remains the controller's). Vote and PreVote freshness become the `raft.tla` `(lastLogTerm, lastLogIndex)` lexicographic comparison. The change is a pure, synchronous, deterministic addition to the core; all durability/I/O stays in adapters.
|
||||
|
||||
### Goals
|
||||
- Add a library-owned `commit_index` and `raft_advance_commit(...)` implementing `raft.tla` `AdvanceCommitIndex` (line 259) **with** the current-term rule `log[Max(agreeIndexes)].term == currentTerm` (line 270) — the Figure-8 safety rule (G4).
|
||||
- Add the required callback `on_commit_advanced(ctx, commit_index)` that **triggers** the existing synchronous commit barrier; the library never performs the barrier itself.
|
||||
- Upgrade vote freshness to the `raft.tla` `HandleRequestVoteRequest` `logOk` `(lastLogTerm, lastLogIndex)` comparison (line 285) on both the binding vote path and the PreVote eligibility path (G6).
|
||||
- Extend `raft_sim.c` to assert the **Figure-8** refusal and **Leader Completeness** across forced leader changes.
|
||||
|
||||
### Non-Goals
|
||||
- **No log model / AppendEntries / per-peer progress** — owned by `raft-log-replication` and reused verbatim (`log_len`, `next_index[]`, `match_index[]`, `on_get_entry_term`, `on_persist_entry`, `on_send_append_entries`, `on_truncate_from`, `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`).
|
||||
- **No commit barrier / durability mechanism** — the synchronous barrier, fsync/group-commit, persist-before-ACK, reply gate, and snapshot transport stay in the controller (`ha-term-safety` adapter). This spec only triggers the barrier.
|
||||
- **No config-change markers / upgrade snapshot** — owned by `ha-term-safety`.
|
||||
- **No conformance / TLC trace-check harness** — owned by `raft-sim-conformance`.
|
||||
- **No controller wiring** — replacing `ha_compute_safe_committed_lsn`'s term-blind advance with the library's `commit_index` is `ha-term-safety`; this spec defines only the library entry point + callback it will route through.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
- The library `commit_index` field in `raft_state_t` and its restore field in `raft_config_t` (`initial_commit_index`, default 0).
|
||||
- The new entry point `raft_advance_commit(raft_state_t *s)` implementing `AdvanceCommitIndex` with the current-term rule.
|
||||
- The new required callback `on_commit_advanced` in `raft_callbacks_t` (fired only on strict advance) + its `raft_init` validation.
|
||||
- The `(lastLogTerm, lastLogIndex)` freshness upgrade on the binding vote path (`raft_on_vote_request`) and the PreVote eligibility path (`raft_on_pre_vote_request`), including carrying `last_log_term` alongside `last_log_index`.
|
||||
- Simulator extensions in `raft_sim.c` + tests in `src/lib/testsuite/test_raft.c` asserting Figure-8 refusal and Leader Completeness.
|
||||
|
||||
### Out of Boundary
|
||||
- The log model, AppendEntries, and per-peer `match_index[]` themselves (`raft-log-replication`) — consumed verbatim, never redefined.
|
||||
- The synchronous commit barrier, fsync/group-commit, persist-before-ACK durability, reply gate, snapshot transport (controller / `ha-term-safety`).
|
||||
- Config markers in the unified log + the change-boundary snapshot (`ha-term-safety`).
|
||||
- The conformance / TLC trace-check harness (`raft-sim-conformance`).
|
||||
- The controller's replacement of `ha_compute_safe_committed_lsn` with `commit_index` and the controller's freshness-floor plumbing of `last_log_term` onto the wire (`ha-term-safety`). This spec specifies only **what the library entry point/callback route**, not the controller mechanics.
|
||||
|
||||
### Allowed Dependencies
|
||||
- The implemented election core (`raft-library`) and the log half (`raft-log-replication`): `raft_state_t`, `raft_callbacks_t`, `raft_init`, the vote/PreVote path, `log_len`, `next_index[]`, `match_index[]`, `on_get_entry_term`. All reused; the vote path is extended additively.
|
||||
- `errno.h`, `stdint.h`, `string.h` only — the core stays free-standing for the simulator.
|
||||
- The simulator harness (`raft_sim.c`) for verification.
|
||||
- Must NOT depend on any controller header, WAL type, capnp type, or transport — the core stays pure.
|
||||
|
||||
### Revalidation Triggers
|
||||
Dependent specs (`raft-sim-conformance`, `ha-term-safety`) must re-check integration if any of the following change:
|
||||
- The signature or semantics of `raft_advance_commit` or the `on_commit_advanced` callback (especially the strict-advance / fire-once rule and "trigger only, never performs the barrier").
|
||||
- The placement of the current-term rule (`on_get_entry_term(Max(agreeIndexes)) == currentTerm`) or the `agreeIndexes`/quorum definition.
|
||||
- The freshness signature change — the addition of `last_log_term` to `raft_on_vote_request`, the PreVote eligibility path, and any vote-request payload / `on_vote_eligible` extension.
|
||||
- The `commit_index` restore convention (`initial_commit_index`, default 0) or its monotonicity guarantee.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
After `raft-log-replication`, the library owns the log as pure, synchronous, deterministic state: per-entry term via `on_get_entry_term`, per-peer `next_index[]` / `match_index[]`, and `leader_commit` carried **opaquely** (never acted on). The election core delegates freshness to `on_vote_eligible`, which today receives only `(from_idx, term)` — the caller checks `config_tag` / applied-LSN floor but has no last-log **term**. This feature follows the same discipline:
|
||||
|
||||
- **Pure / synchronous / deterministic core** — `raft_advance_commit` reads entry terms via `on_get_entry_term` and emits a single `on_commit_advanced` notification; it owns no entry bytes and performs no I/O. The freshness upgrade is a predicate computed from values already in the library log model plus the candidate's advertised `(last_log_term, last_log_index)`.
|
||||
- **Persist-before-act ordering** — unchanged on the vote path: a vote that passes the new `(lastLogTerm, lastLogIndex)` predicate is still persisted via `on_persist` before being granted. `commit_index` advance is not itself persisted by the library — durability of the committed position remains the controller barrier's job, triggered by `on_commit_advanced`.
|
||||
- **Status returns, never self-terminate** — `raft_advance_commit` returns a status; no new path aborts.
|
||||
- **Separate actions** — mirroring `raft.tla`, `AdvanceCommitIndex` is a **distinct** action from `HandleAppendEntriesResponse`. `raft_on_append_entries_reply` (owned upstream) only updates `match_index[]`; commit advance is the separate `raft_advance_commit` call. This keeps atomic regions small and lets a single-node leader commit (the `{self}` term in `agreeIndexes`).
|
||||
|
||||
### Architecture Pattern & Boundary Map
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph PureCore["src/lib/raft/raft.c — PURE CORE (this spec)"]
|
||||
ADV["raft_advance_commit()<br/>AdvanceCommitIndex (tla:259)<br/>+ current-term rule (tla:270)"]
|
||||
VOTE["raft_on_vote_request()<br/>logOk (lastLogTerm,lastLogIndex) (tla:285) G6"]
|
||||
PVOTE["raft_on_pre_vote_request()<br/>same (lastLogTerm,lastLogIndex) freshness"]
|
||||
end
|
||||
subgraph Upstream["raft-log-replication — REUSED VERBATIM (not this spec)"]
|
||||
MI["match_index[] / log_len"]
|
||||
GET["on_get_entry_term(index) -> term|0"]
|
||||
REPLY["raft_on_append_entries_reply()<br/>updates match_index[] only"]
|
||||
end
|
||||
subgraph CBs["raft_callbacks_t — NEW seam (this spec)"]
|
||||
C5["on_commit_advanced(ctx, commit_index)<br/>fired ONLY on strict advance"]
|
||||
end
|
||||
subgraph Consumers["ADAPTERS — implement callback (NOT this spec's heavy wiring)"]
|
||||
CTRL["controller adapter (ha-term-safety):<br/>on_commit_advanced -> EXISTING sync commit barrier<br/>(persist-before-ACK preserved, NOT replaced)"]
|
||||
SIM["raft_sim.c: drive raft_advance_commit,<br/>assert Figure-8 + Leader Completeness (this spec)"]
|
||||
end
|
||||
REPLY -.updates.-> MI
|
||||
ADV --> MI & GET
|
||||
ADV --> C5
|
||||
VOTE --> GET
|
||||
PVOTE --> GET
|
||||
C5 --> CTRL
|
||||
C5 --> SIM
|
||||
ADV --> SIM
|
||||
```
|
||||
|
||||
**Architecture Integration**:
|
||||
- **Selected pattern**: Pure state machine + callback ports (Hexagonal/Ports-and-Adapters), already established by the election and log cores. Commit advance is a new pure action; `on_commit_advanced` is a new port adapters implement.
|
||||
- **Domain boundaries**: the commit-safety **decision** (which index is safe to commit, given quorum + current-term) stays in the library; the durable **act** (the synchronous commit barrier, fsync, ACK) stays in the controller adapter, triggered by the port.
|
||||
- **Existing patterns preserved**: `raft_config_t` extensibility, persist-before-act on the vote path, NULL-checked required callbacks, status-code returns, `XCE_TESTING`-gated simulator.
|
||||
- **Steering compliance**: matches `src/lib/CLAUDE.md` (free-standing C, gnu17), `ha-async-io.md` (no blocking in the core; the barrier/async lives in the adapter), `testing.md` (TDD: simulator/test first), `ha-async-io.md` durability policy ("persist-before-ACK + synchronous commit barrier" preserved — the callback triggers it), "controller never self-terminates" (library returns statuses only).
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice / Version | Role in Feature | Notes |
|
||||
|-------|------------------|-----------------|-------|
|
||||
| Library core | C (gnu17), `errno.h`/`stdint.h`/`string.h` only | Pure commit-safety + freshness | Free-standing; no controller/capnp deps |
|
||||
| Callbacks | `raft_callbacks_t` function pointers | Commit-advance trigger seam | 1 new required callback (`on_commit_advanced`) |
|
||||
| Verification | `raft_sim.c` (XCE_TESTING) + `test_raft.c` | Figure-8 + Leader Completeness | Reuses the in-memory log backing from `raft-log-replication` |
|
||||
| Adapter (downstream) | controller commit barrier (`ha-term-safety`) | Durable commit / ACK | Not built here; only triggered |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Modified Files
|
||||
- `src/lib/raft/raft.h` — Add `uint64_t commit_index;` to `raft_state_t`; add `uint64_t initial_commit_index;` to `raft_config_t`; add the `on_commit_advanced` callback to `raft_callbacks_t`; declare `raft_advance_commit`; extend the vote-path signatures to carry `last_log_term` (alongside the existing `last_log_index`) per the chosen freshness-plumbing option below; add a Figure-8 / Leader-Completeness driver + assertion helper to the simulator section.
|
||||
- `src/lib/raft/raft.c` — Implement `raft_advance_commit` (agreeIndexes + current-term rule + monotonic strict-advance firing `on_commit_advanced`); add the `on_commit_advanced` NULL check + `initial_commit_index` restore in `raft_init`; upgrade the freshness predicate in `raft_on_vote_request` and `raft_on_pre_vote_request` to the `(lastLogTerm, lastLogIndex)` comparison.
|
||||
- `src/lib/raft/raft_sim.c` — Drive `raft_advance_commit` after delivery; the Figure-8 construction; the Leader-Completeness assertion across forced leader changes; the freshness-gate exercise.
|
||||
- `src/lib/testsuite/test_raft.c` — New tests for `raft_advance_commit` (current-term hold/advance, monotonicity, non-leader), `on_commit_advanced` fire-once/never semantics, the `(lastLogTerm, lastLogIndex)` vote/PreVote freshness, and the simulator invariants (TDD-first).
|
||||
|
||||
### Directory Structure (unchanged)
|
||||
```
|
||||
src/lib/raft/
|
||||
├── raft.h # + commit_index, on_commit_advanced, raft_advance_commit, last_log_term on vote path
|
||||
├── raft.c # + AdvanceCommitIndex(current-term rule), (lastLogTerm,lastLogIndex) freshness
|
||||
└── raft_sim.c # + commit-advance driver, Figure-8, Leader Completeness assert
|
||||
src/lib/testsuite/
|
||||
└── test_raft.c # + commit-safety + freshness + Figure-8/Leader-Completeness tests
|
||||
```
|
||||
|
||||
> Each file keeps one responsibility: `raft.c` = pure protocol, `raft_sim.c` = deterministic harness, `test_raft.c` = assertions. No new file is introduced; the existing module pattern is extended.
|
||||
|
||||
## System Flows
|
||||
|
||||
### Commit advance: separate action after replication (raft.tla AdvanceCommitIndex)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant L as Leader (raft.c)
|
||||
participant LC as Leader adapter (cb)
|
||||
Note over L: replication already updated match_index[]<br/>via raft_on_append_entries_reply (raft-log-replication)
|
||||
Note over L: raft_advance_commit() — separate action (tla:259)
|
||||
L->>L: agreeIndexes = { i in 1..log_len :<br/>{self} ∪ {p: match_index[p]>=i} ∈ Quorum }
|
||||
alt agreeIndexes empty
|
||||
Note over L: hold commit_index (else-branch)
|
||||
else Max(agreeIndexes) from a PREVIOUS term
|
||||
L->>LC: on_get_entry_term(Max) != currentTerm
|
||||
Note over L: hold commit_index (Figure-8 refusal, G4)
|
||||
else Max(agreeIndexes) from CURRENT term
|
||||
L->>LC: on_get_entry_term(Max) == currentTerm
|
||||
Note over L: commit_index = Max(agreeIndexes) (strict advance)
|
||||
L->>LC: on_commit_advanced(ctx, commit_index)
|
||||
Note over LC: trigger EXISTING sync commit barrier<br/>(persist-before-ACK, NOT replaced)
|
||||
end
|
||||
```
|
||||
|
||||
### Vote freshness: (lastLogTerm, lastLogIndex) comparison (raft.tla HandleRequestVoteRequest logOk)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant V as Voter (raft.c)
|
||||
participant VC as Voter adapter (cb)
|
||||
Note over V: raft_on_vote_request(..., cand_last_log_term, cand_last_log_index)
|
||||
Note over V: UpdateTerm / stale-term / one-vote-per-term (unchanged)
|
||||
V->>VC: on_get_entry_term(log_len) -> local_last_log_term (0 if empty)
|
||||
Note over V: logOk = cand_last_log_term > local_last_log_term<br/>∨ (== ∧ cand_last_log_index >= log_len)
|
||||
alt !logOk
|
||||
Note over V: DENY (even if all other gates would grant) — G6
|
||||
else logOk AND application gate AND invariants pass
|
||||
V->>VC: on_persist(term, from_idx) %% persist-before-grant unchanged
|
||||
Note over V: GRANT
|
||||
end
|
||||
```
|
||||
|
||||
Flow-level decisions:
|
||||
- **Commit is a separate action** — `match_index[]` is updated by `raft_on_append_entries_reply` (upstream); the commit index advances only when `raft_advance_commit` is driven, exactly as `raft.tla` separates the two actions (R2.6). This lets a single-node leader commit (`{self}` ∈ `agreeIndexes`).
|
||||
- **Current-term rule is the Figure-8 gate** — `commit_index` advances to `Max(agreeIndexes)` only when that entry is from `currentTerm`; a previous-term entry replicated to a quorum is **held** until a current-term entry above it commits, then committed indirectly (R1.3/R1.4, closes G4).
|
||||
- **`on_commit_advanced` fires only on strict advance** — never on a held commit index, never from `raft_on_append_entries_reply` (R2.2/R2.3/R2.6).
|
||||
- **Freshness denies before any grant** — a stale `(lastLogTerm, lastLogIndex)` denies even when the term/vote invariants and the application gate would grant (R3.4, closes G6); persist-before-grant is unchanged (R3.6).
|
||||
|
||||
## Freshness Plumbing — design decision
|
||||
|
||||
`raft.tla` `logOk` needs the candidate's `(lastLogTerm, lastLogIndex)` and the voter's own. Two options were considered for carrying the candidate values into the library:
|
||||
|
||||
- **Option A — extend the `on_vote_eligible` callback signature** to `on_vote_eligible(ctx, from_idx, term, last_log_term, last_log_index)` and have the library compute `logOk` against `on_get_entry_term(log_len)`/`log_len` before calling the gate. **Rejected as the sole mechanism**: it pushes the protocol-defining `logOk` either into the adapter (violating "library owns protocol decisions") or splits it.
|
||||
- **Option B (chosen) — the library owns `logOk`.** The candidate's `last_log_term` and `last_log_index` are accepted as explicit arguments on `raft_on_vote_request` (and on the PreVote eligibility path), the library computes `logOk` against its own `on_get_entry_term(log_len)` / `log_len`, and only then consults the unchanged application gate `on_vote_eligible` for XCE-specific policy (primary-deny / heartbeat stickiness / catch-up). This keeps the `raft.tla` `logOk` predicate **inside** the library (R3.1, R5.5) while leaving `on_vote_eligible` for application policy that is genuinely outside Raft.
|
||||
|
||||
The candidate advertises its `(last_log_term, last_log_index)` in the `RequestVote` / PreVote wire RPC; the controller adapter decodes them and passes them to the library entry point. The on-wire encoding of `last_log_term` (and the controller's freshness-floor plumbing) is the `ha-term-safety` adapter's job — out of this spec's scope; this spec fixes only the library signatures and the predicate.
|
||||
|
||||
> **Note — freshness-decision locus moves from caller/dispatcher into the library (both paths).** Today the freshness decision lives **outside** the library: `raft.h` (lines 246–248) documents that PreVote freshness "is applied by the dispatcher before this call", and the binding vote path delegates the decision to `on_vote_eligible`. Option B **moves that decision locus into the library** for **both** the binding (`raft_on_vote_request`) and the PreVote (`raft_on_pre_vote_request`) paths: `raft.c` now computes the `(lastLogTerm, lastLogIndex)` `logOk` itself rather than relying on the caller/dispatcher. This **supersedes the existing `raft.h` lines 246–248 doc contract**, and updating those `raft.h` doc comments (so the header no longer states the dispatcher owns PreVote freshness) is part of the header-contract task.
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Design Elements | raft.tla action |
|
||||
|-------------|---------|-----------------|-----------------|
|
||||
| 1.1–1.7 | Library `commit_index`, agreeIndexes, current-term rule, monotonic, leader-only | `raft_state_t.commit_index`; `raft_advance_commit()`; `on_get_entry_term`; `match_index[]` (upstream) | `AdvanceCommitIndex` (259), current-term rule (270) |
|
||||
| 2.1–2.6 | `on_commit_advanced` trigger, fire-once, separate from reply | `raft_callbacks_t.on_commit_advanced`; strict-advance firing; `raft_init` validation | (seam; `AdvanceCommitIndex` distinct from `HandleAppendEntriesResponse`) |
|
||||
| 3.1–3.6 | `(lastLogTerm,lastLogIndex)` binding-vote freshness | `raft_on_vote_request` logOk; `on_get_entry_term(log_len)`/`log_len`; persist-before-grant | `HandleRequestVoteRequest` logOk (285) |
|
||||
| 4.1–4.5 | Same freshness on PreVote, side-effect-free | `raft_on_pre_vote_request` logOk; no persist/term/role mutation | `HandleRequestVoteRequest` logOk (285), PreVote extension |
|
||||
| 5.1–5.4 | Barrier/durability preserved as policy; PreVote/priority unchanged; no abort | `on_commit_advanced` as trigger only; additive-only changes | (policy; not a TLA+ action) |
|
||||
| 6.1–6.6 | Simulator Figure-8 + Leader Completeness + freshness gate | `raft_sim.c` commit driver, Figure-8 construction, Leader-Completeness assert, freshness exercise | Figure-8 / Leader Completeness invariant |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|
||||
|-----------|-------|--------|--------------|------------------|-----------|
|
||||
| `commit_index` (`raft_state_t`) | Library core | Library-owned commit position | 1 | log core (P0) | State |
|
||||
| `raft_advance_commit` | Library core | AdvanceCommitIndex + current-term rule | 1, 2 | `on_get_entry_term`, `match_index[]`, `on_commit_advanced` (P0) | Service |
|
||||
| `on_commit_advanced` callback | Seam | Commit-advance trigger | 2, 5 | controller barrier / sim (P0) | Service |
|
||||
| `raft_on_vote_request` freshness | Library core | `(lastLogTerm,lastLogIndex)` logOk | 3, 5 | `on_get_entry_term`, `log_len`, `on_persist`, `on_vote_eligible` (P0) | Service |
|
||||
| `raft_on_pre_vote_request` freshness | Library core | Same logOk, side-effect-free | 4, 5 | `on_get_entry_term`, `log_len`, `on_vote_eligible` (P0) | Service |
|
||||
| Simulator | Verification | Figure-8 + Leader Completeness | 6 | all of the above (P0) | State |
|
||||
|
||||
### Library Core
|
||||
|
||||
#### `commit_index` addition to `raft_state_t`
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Hold the library's view of the safe commit position (TLA+ `commitIndex`) |
|
||||
| Requirements | 1.1, 1.5 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `uint64_t commit_index;` — the highest 1-based index (== WAL LSN) the library considers safely committed. Monotonic non-decreasing.
|
||||
- Restored at init from the adapter (`raft_config_t.initial_commit_index`, default 0; the library never reads storage — consistent with `initial_term` / `initial_log_len`).
|
||||
- The library never advances `commit_index` except through `raft_advance_commit`; `raft_on_append_entries_reply` (upstream) leaves it untouched (R2.6).
|
||||
|
||||
#### `raft_advance_commit` — AdvanceCommitIndex (raft.tla:259) with the current-term rule (line 270)
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Advance `commit_index` to `Max(agreeIndexes)` only when that entry is from `currentTerm` |
|
||||
| Requirements | 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.2, 2.3 |
|
||||
|
||||
##### Service Interface
|
||||
```c
|
||||
/*
|
||||
* raft_advance_commit: leader recomputes its commit index (raft.tla
|
||||
* AdvanceCommitIndex, 259). A SEPARATE action from append-entries reply
|
||||
* handling (mirrors raft.tla, so a single-node leader can commit and to
|
||||
* keep atomic regions small).
|
||||
*
|
||||
* Computes agreeIndexes = { index in 1..log_len :
|
||||
* ({self} ∪ {peer : match_index[peer] >= index}) is a quorum }.
|
||||
* If agreeIndexes is non-empty AND on_get_entry_term(Max(agreeIndexes))
|
||||
* == s->term, sets commit_index = Max(agreeIndexes) (Figure-8 current-term
|
||||
* rule) and, on a STRICT advance, fires on_commit_advanced(ctx, commit_index)
|
||||
* exactly once.
|
||||
* Otherwise leaves commit_index unchanged and does NOT fire on_commit_advanced.
|
||||
*
|
||||
* The library performs NO durability work here — on_commit_advanced only
|
||||
* TRIGGERS the controller's existing synchronous commit barrier.
|
||||
*
|
||||
* Returns RAFT_OK on success (whether or not the index advanced);
|
||||
* -EPERM if role != RAFT_LEADER (commit_index left unchanged).
|
||||
*/
|
||||
int raft_advance_commit(raft_state_t *s);
|
||||
```
|
||||
- Preconditions: `s->role == RAFT_LEADER` (else `-EPERM`, no change — R1.6).
|
||||
- Postconditions: `commit_index` either unchanged or advanced to `Max(agreeIndexes)` when that entry is current-term; `on_commit_advanced` fired iff `commit_index` strictly increased.
|
||||
- Invariants: `commit_index` never decreases (R1.5); the library reads entry terms only via `on_get_entry_term` (R1.7); `commit_index ≤ log_len`.
|
||||
|
||||
> **Quorum**: `(n_peers / 2) + 1`, identical to the election-core quorum; `match_index[self]` is implicitly `log_len` (self always agrees up to its own log length), realized by seeding the agree-count with self for each candidate index. `agreeIndexes` is computed by scanning candidate indexes from `log_len` downward and taking the first that reaches quorum (equivalent to `Max(agreeIndexes)` without materializing the set).
|
||||
|
||||
#### Freshness upgrade — `raft_on_vote_request` (raft.tla:285, closes G6)
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Add the `(lastLogTerm, lastLogIndex)` lexicographic logOk before the application gate |
|
||||
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 |
|
||||
|
||||
##### Service Interface
|
||||
```c
|
||||
/*
|
||||
* raft_on_vote_request: process an incoming RequestVote RPC (extended).
|
||||
* Adds the raft.tla HandleRequestVoteRequest logOk (285): the library
|
||||
* computes
|
||||
* logOk = (cand_last_log_term > LastTerm(local))
|
||||
* || (cand_last_log_term == LastTerm(local)
|
||||
* && cand_last_log_index >= log_len)
|
||||
* where LastTerm(local) = (log_len > 0) ? on_get_entry_term(log_len) : 0.
|
||||
* Step order (unchanged except the new logOk gate):
|
||||
* 1. stale-term deny / UpdateTerm (unchanged)
|
||||
* 2. role gate, one-vote-per-term (unchanged)
|
||||
* 3. logOk gate (NEW, G6): deny if !logOk, before on_vote_eligible
|
||||
* 4. application gate on_vote_eligible (unchanged)
|
||||
* 5. on_persist before grant (unchanged)
|
||||
* Returns RAFT_GRANT / RAFT_DENY; sets *reply_term.
|
||||
*/
|
||||
int raft_on_vote_request(raft_state_t *s, int from_idx, uint64_t term,
|
||||
uint64_t cand_last_log_term,
|
||||
uint64_t cand_last_log_index,
|
||||
uint64_t *reply_term);
|
||||
```
|
||||
- Preconditions: none beyond a valid `s`.
|
||||
- Postconditions: a vote is granted only when the existing invariants, the new `(lastLogTerm, lastLogIndex)` logOk, and the application gate all pass; persisted before grant.
|
||||
- Invariants: `logOk` is evaluated **before** `on_vote_eligible`; a `!logOk` denies regardless of application policy (R3.4); no weakening of stale-term / one-vote-per-term (R3.5).
|
||||
|
||||
> **Signature change is a Revalidation Trigger.** `raft_on_vote_request` gains `cand_last_log_term` and `cand_last_log_index`. The controller and the simulator (the two callers) must pass the candidate's advertised last-log `(term, index)`. The PreVote path mirrors this.
|
||||
|
||||
#### Freshness upgrade — `raft_on_pre_vote_request` (side-effect-free)
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Apply the identical `(lastLogTerm, lastLogIndex)` logOk on the straw poll, mutating nothing |
|
||||
| Requirements | 4.1, 4.2, 4.3, 4.4, 4.5 |
|
||||
|
||||
##### Service Interface
|
||||
```c
|
||||
/*
|
||||
* raft_on_pre_vote_request: answer a PreVote probe (extended).
|
||||
* Applies the SAME (lastLogTerm, lastLogIndex) logOk as the binding vote
|
||||
* (Requirement 3.1), in addition to the existing stale-term deny and the
|
||||
* on_vote_eligible gate. Mutates NOTHING: no term/voted_for/role/commit_index
|
||||
* change, no on_persist (PreVote invariant preserved).
|
||||
* Returns RAFT_GRANT / RAFT_DENY.
|
||||
*/
|
||||
int raft_on_pre_vote_request(raft_state_t *s, int from_idx, uint64_t cand_term,
|
||||
uint64_t cand_last_log_term,
|
||||
uint64_t cand_last_log_index);
|
||||
```
|
||||
- Preconditions: none beyond a valid `s`.
|
||||
- Postconditions: grant only when stale-term, logOk, and the application gate all pass; no persistent state touched.
|
||||
- Invariants: identical freshness rule to the binding path (R4.1, R4.2); side-effect-free (R4.4).
|
||||
|
||||
### Seam — `raft_callbacks_t` addition (verbatim Authoritative shared contract)
|
||||
|
||||
Appended to `raft_callbacks_t` and validated **required** in `raft_init`. Reproduced verbatim from the brief so downstream specs depend on the exact shape:
|
||||
|
||||
```c
|
||||
/* REQUIRED: fired AFTER commit_index strictly advances. The controller adapter
|
||||
* feeds this into the EXISTING synchronous commit barrier / persist-before-ACK
|
||||
* durability — this callback TRIGGERS commit-eval; it does NOT replace the
|
||||
* barrier. */
|
||||
void on_commit_advanced(void *ctx, uint64_t commit_index);
|
||||
```
|
||||
|
||||
**Contract rules (R2.4, R5.1)**: the library decides *whether* the commit index advanced; the callback only triggers the adapter's commit work. It is fired exactly once per strict advance (R2.2), never on a held commit index (R2.3), and never from `raft_on_append_entries_reply` (R2.6). It is NULL-checked and required in `raft_init` (`-EINVAL` if NULL — R2.5), matching how the other required callbacks are validated.
|
||||
|
||||
### Controller adapter (seam only — heavy wiring is `ha-term-safety`)
|
||||
|
||||
This spec describes the adapter **only at the seam**; the full controller integration is the downstream `ha-term-safety` consumer.
|
||||
|
||||
- `on_commit_advanced` → the controller routes the new safe `commit_index` into its **existing** synchronous commit barrier (commit-eval sweep, persist-before-ACK, reply gate) — replacing the term-blind `ha_compute_safe_committed_lsn` advance with the library's current-term-gated value. The barrier mechanism is unchanged; only its **input** (the safe commit position) now comes from the library.
|
||||
- The candidate's advertised `(last_log_term, last_log_index)` is decoded from the `RequestVote` / PreVote wire RPC and passed to `raft_on_vote_request` / `raft_on_pre_vote_request`; `last_log_term` is read back from the WAL term (`on_get_entry_term`-backed index) on recovery. The on-wire encoding and the freshness-floor plumbing are `ha-term-safety`.
|
||||
- On recovery, the controller restores `commit_index` (= recovered committed LSN) into `raft_config_t.initial_commit_index`.
|
||||
|
||||
## Data Models
|
||||
|
||||
### Domain Model
|
||||
- **Commit position (logical)**: `commit_index : uint64` — the highest 1-based index (== WAL LSN) safely committed, owned by the library, monotonic.
|
||||
- **Convention**: callers pass `s->commit_index` as the `leader_commit` argument to `raft_send_append_entries` — the leader's committed value rides AppendEntries from `commit_index`, which this spec owns (upstream `raft-log-replication` carries `leader_commit` opaquely; the value it carries is the `commit_index` defined here).
|
||||
- **Freshness tuple**: `(last_log_term : uint64, last_log_index : uint64)` per candidate, compared lexicographically against the voter's own `(on_get_entry_term(log_len), log_len)`.
|
||||
|
||||
### Logical Data Model
|
||||
- `raft_state_t` gains `commit_index`.
|
||||
- `raft_config_t` gains `initial_commit_index` (restored committed LSN at boot; 0 on first boot — the library never reads storage, consistent with `initial_term` / `initial_log_len`).
|
||||
- No new persisted on-disk format: `commit_index` durability is the controller barrier's existing responsibility; the library imposes no new persisted layout. `last_log_term` reuses the WAL `hdr.term` already written by `raft-log-replication`'s adapter.
|
||||
|
||||
### Data Contracts & Integration
|
||||
- **Freshness tuple**: `raft_on_vote_request` and `raft_on_pre_vote_request` accept `(cand_last_log_term, cand_last_log_index)` as explicit arguments. The voter's own values come from `on_get_entry_term(log_len)` (term `0` when `log_len == 0`, consistent with `raft.tla` `LastTerm`) and `log_len`.
|
||||
- **Commit trigger**: `on_commit_advanced(ctx, commit_index)` carries only the new committed index; no entry bytes, no term — the adapter looks up whatever it needs through its own state.
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Strategy
|
||||
The library returns status codes only and **never aborts** (preserves "controller never self-terminates"). Callbacks for commit-advance are trigger-only and return void; the freshness path keeps the existing persist-failure handling.
|
||||
|
||||
### Error Categories and Responses
|
||||
- **Caller misuse**: `raft_advance_commit` while not leader → `-EPERM`, `commit_index` unchanged, no `on_commit_advanced`. NULL `on_commit_advanced` at `raft_init` → `-EINVAL`.
|
||||
- **Held commit (Figure-8)**: when `Max(agreeIndexes)` is a previous-term entry, `commit_index` is held — this is the correct safety behavior, not an error; no callback fires.
|
||||
- **Empty agreeIndexes**: no quorum agrees on any index → hold, no callback.
|
||||
- **Stale freshness**: `!logOk` → vote/PreVote DENY (not an error; the standard freshness rejection).
|
||||
- **Persist failure on grant**: unchanged from the existing vote path — a failed `on_persist` denies the vote without recording it.
|
||||
|
||||
### Monitoring
|
||||
Reuse the optional `on_trace` callback for new events: `"commit_advance"`, `"commit_hold_prevterm"`, `"commit_hold_noquorum"`, `"vote_denied_log_term"` (stale `lastLogTerm`), `"vote_denied_log_index"` (tie term, stale index), `"pre_vote_denied_freshness"`. NULL-checked exactly like existing trace calls.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
TDD-mandatory (`testing.md`): tests and simulator drivers are written **before** the `raft.c` implementation (RED → GREEN). All tests are `XCE_TESTING`-gated and live in `src/lib/testsuite/test_raft.c` driving `raft_sim.c`. They reuse the in-memory per-node log backing added by `raft-log-replication`.
|
||||
|
||||
### Unit Tests (entry points, against the in-memory sim log)
|
||||
1. `raft_advance_commit` current-term advance: a leader whose `match_index[]` shows a quorum agreeing through index `k`, with the entry at `k` from `currentTerm`, advances `commit_index` to `k` and fires `on_commit_advanced(k)` once (R1.2, R1.3, R2.2).
|
||||
2. `raft_advance_commit` Figure-8 hold: a leader with a quorum agreeing through a **previous-term** entry at `k` (no current-term entry above it) holds `commit_index` and fires nothing (R1.4, R2.3 — G4).
|
||||
3. `raft_advance_commit` monotonicity + non-leader: never decreases `commit_index`; returns `-EPERM` with no change and no callback when not leader (R1.5, R1.6).
|
||||
4. `on_commit_advanced` separation: `raft_on_append_entries_reply` updating `match_index[]` does **not** by itself advance `commit_index` or fire `on_commit_advanced`; only a subsequent `raft_advance_commit` does (R2.6).
|
||||
5. `raft_init` requires `on_commit_advanced`: `-EINVAL` when NULL; restores `initial_commit_index` (R2.5, R1.1).
|
||||
6. `(lastLogTerm, lastLogIndex)` vote freshness: a candidate with a stale `last_log_term` is denied even with a greater `last_log_index`; a candidate with a fresher `last_log_term` is granted with a smaller `last_log_index`; a tie on term grants only when `last_log_index >= log_len` (R3.1, R3.4). PreVote applies the identical rule with no persist/term/role mutation (R4.1, R4.4).
|
||||
|
||||
### Integration / Simulator Tests (Figure-8, Leader Completeness)
|
||||
1. **Figure-8 scenario**: construct a previous-term entry replicated to a quorum but not yet covered by a committed current-term entry; drive `raft_advance_commit` and assert `commit_index` does **not** move over it; then replicate a current-term entry above it to a quorum, drive `raft_advance_commit`, and assert `commit_index` advances over **both** (indirect commit) (R6.2, R6.3 — G4).
|
||||
2. **Leader Completeness**: across forced leader changes (timeout / partition-driven), assert that any entry committed (≤ `commit_index` on the committing leader) is present in the log of every node that subsequently becomes leader (R6.4) — the `(lastLogTerm, lastLogIndex)` freshness gate (R6.5) is what makes this hold, so the two tests are complementary.
|
||||
3. **Freshness gate exercise**: drive votes with mismatched `(last_log_term, last_log_index)` tuples and assert the lexicographic outcome (stale-term-but-higher-index denied; fresher-term-but-lower-index granted) (R6.5).
|
||||
4. **Trigger semantics**: assert `on_commit_advanced` fires exactly on strict advance and not while the current-term rule holds the commit index (R6.6).
|
||||
112
.kiro/specs/raft-commit-safety/requirements.md
Normal file
112
.kiro/specs/raft-commit-safety/requirements.md
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
`src/lib/raft/raft.{c,h}` is the pure, callback-driven Raft library for XCE. The `raft-library` spec gave it the **election half** of `raft/raft.tla` (Ongaro's canonical Raft TLA⁺), and the upstream `raft-log-replication` spec gave it the **log half**: a library-owned log where every entry carries a `uint64_t` term (1-based index = WAL LSN), `ClientRequest` append (`raft_log_append`), `AppendEntries` emission/handling, per-peer `next_index[]` / `match_index[]`, the `on_get_entry_term` / `on_persist_entry` / `on_send_append_entries` / `on_truncate_from` callbacks, and `UpdateTerm` on every RPC. That upstream spec deliberately carries `leader_commit` **opaquely** through `on_send_append_entries` and `raft_on_append_entries_request` **without acting on it** — it never advances a commit index.
|
||||
|
||||
This feature closes the remaining two `raft.tla` **safety** gaps from the gap analysis (`.kiro/specs/ha-term-safety/TLA-gap-analysis.md`):
|
||||
|
||||
- **G4 — term-blind commit advance.** `raft.tla` `AdvanceCommitIndex` (line 259) advances `commitIndex` to `Max(agreeIndexes)` **only if `log[Max(agreeIndexes)].term == currentTerm`** — the Figure-8 safety rule: a leader must never mark a previous-term entry committed by replica-count alone; it commits the older entry only indirectly, once an entry from its **own** current term reaches a quorum. XCE's commit barrier + `ha_compute_safe_committed_lsn` advanced `committed` term-blind ("from 1 granted voter") because the WAL term was dead (G1); the own-term `WAL_NOOP` gate is only a partial mitigation. This spec adds `raft_advance_commit(...)` to the library, computing `agreeIndexes` from per-peer `match_index[]` and advancing `commit_index` **only when the entry at the max agreed index is from the current term**, with a new `on_commit_advanced(ctx, commit_index)` callback that **triggers** the existing synchronous commit barrier.
|
||||
|
||||
- **G6 — LSN-only vote freshness.** `raft.tla` `HandleRequestVoteRequest` `logOk` (line 285) compares the candidate's `(lastLogTerm, lastLogIndex)` against the voter's **lexicographically**: grant only if the candidate's last log term is strictly greater, or the terms tie and the candidate's last log index is at least the voter's. XCE's `on_vote_eligible` gate uses `config_tag ≥` plus an applied/durable-LSN floor **only** — it never compares `lastLogTerm`. With G6 a node whose log has a higher term but a lower index can be denied (correct) or, worse, a stale-term-but-higher-index candidate can be wrongly granted. This spec upgrades the freshness gate on **both** the binding vote path (`raft_on_vote_request`) and the PreVote eligibility path to carry and compare `last_log_term` alongside the existing `last_log_index`.
|
||||
|
||||
The library core stays **pure / synchronous / deterministic** (no I/O, no threads, no allocation in the hot path): `raft_advance_commit` reads entry terms only through the existing `on_get_entry_term` callback and reports the new commit index only through `on_commit_advanced`; the controller adapter performs the durable commit work. The feature is verified end-to-end in the deterministic simulator (`raft_sim.c`) against **Leader Completeness** and the Figure-8 scenario, without a live cluster.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**:
|
||||
- `raft_advance_commit(...)`: compute `agreeIndexes` from the library's per-peer `match_index[]` (a quorum agrees, including self), and advance `commit_index` to `Max(agreeIndexes)` **only when `on_get_entry_term(Max(agreeIndexes)) == currentTerm`** (the Figure-8 current-term rule, `raft.tla` `AdvanceCommitIndex` line 259, closes G4). Otherwise hold `commit_index` unchanged.
|
||||
- A new `commit_index` field in the library state and the new required callback `void on_commit_advanced(void *ctx, uint64_t commit_index)` fired **after** `commit_index` strictly advances, so the controller adapter feeds it into the **existing synchronous commit barrier / persist-before-ACK durability** (the callback triggers commit-evaluation; it does not perform or replace the barrier).
|
||||
- `(lastLogTerm, lastLogIndex)` vote freshness (`raft.tla` `HandleRequestVoteRequest` `logOk` line 285, closes G6): carry `last_log_term` alongside `last_log_index` into the binding vote path (`raft_on_vote_request`) and the PreVote eligibility path, and grant only on the lexicographic `(lastLogTerm, lastLogIndex)` comparison.
|
||||
- Simulator coverage in `raft_sim.c`: a **Figure-8** scenario (a previous-term entry replicated to a quorum but not yet committed by the current leader) asserting the library refuses to commit it until a current-term entry is committed; and a **Leader Completeness** assertion (a committed entry is present in every future leader's log) across forced leader changes.
|
||||
|
||||
- **Out of scope** (owned elsewhere):
|
||||
- The library log model + `ClientRequest` / `AppendEntries` / `HandleAppendEntries*` / `UpdateTerm`-on-any-RPC, the per-entry term, and the per-peer `next_index[]` / `match_index[]` themselves → `raft-log-replication` (consumed here verbatim; not redefined).
|
||||
- The synchronous commit-barrier mechanism, fsync/group-commit, persist-before-ACK durability, the reply gate, and snapshot transport → controller (`ha-term-safety` adapter). This spec only **triggers** the barrier via `on_commit_advanced`; it neither implements nor weakens it.
|
||||
- Configuration-change markers in the unified log and the upgrade snapshot → `ha-term-safety`.
|
||||
- The full conformance / TLC trace-check harness → `raft-sim-conformance`.
|
||||
- The replacement of `ha_compute_safe_committed_lsn`'s term-blind advance in the controller with the library's current-term-gated `commit_index` → that controller wiring is `ha-term-safety`; this spec defines only the library entry point and callback the controller will route through.
|
||||
|
||||
- **Adjacent expectations**:
|
||||
- Builds on `raft-log-replication` (HARD dependency): needs the `(term, index)` library log, `on_get_entry_term`, and per-peer `match_index[]`. All of that contract is reused verbatim and not redefined here.
|
||||
- Preserves as policy (must NOT be canonicalized away): the synchronous commit barrier + persist-before-ACK durability are **stronger** than canonical Raft's async apply and remain — `on_commit_advanced` triggers the existing barrier, it does not replace it. PreVote and priority leadership are unchanged in mechanism; only the freshness predicate they consult gains `last_log_term`.
|
||||
- "Controller never self-terminates": the library returns status codes only; it never aborts a process. The controller owns liveness/fencing.
|
||||
- Feeds `raft-sim-conformance` (Leader Completeness oracle) and `ha-term-safety` (the controller integration that routes the WAL/commit barrier through `raft_advance_commit` + `on_commit_advanced`). Closes bug-851 G4/G6.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Library-owned commit index with the current-term advance rule (closes G4)
|
||||
|
||||
**Objective:** As the leader of a Raft cluster, I want my commit index to advance only over an index that a quorum holds **and** whose entry is from my current term, so that a previous-term entry is never marked committed by replica-count alone (Figure-8 safety) and Leader Completeness holds.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Raft Library shall maintain, per node, a `commit_index` (the `raft.tla` `commitIndex`) initialized to the value restored from the adapter at boot (default 0), so the library — not term-blind controller code — owns the commit position.
|
||||
2. When `raft_advance_commit(...)` is called while the node's role is `RAFT_LEADER`, the Raft Library shall compute `agreeIndexes` as the set of indexes `index` in `1..log_len` for which the set `{self} ∪ {peer : match_index[peer] ≥ index}` is a quorum, mirroring `raft.tla` `AdvanceCommitIndex` `agreeIndexes`.
|
||||
3. When `agreeIndexes` is non-empty and `on_get_entry_term(Max(agreeIndexes)) == currentTerm`, the Raft Library shall set `commit_index` to `Max(agreeIndexes)`, mirroring the `raft.tla` `AdvanceCommitIndex` `newCommitIndex` then-branch (the Figure-8 current-term rule).
|
||||
4. If `agreeIndexes` is empty, or `on_get_entry_term(Max(agreeIndexes)) != currentTerm`, then the Raft Library shall leave `commit_index` unchanged, mirroring the `raft.tla` `AdvanceCommitIndex` else-branch (refusing to commit a previous-term entry by replica-count alone).
|
||||
5. The Raft Library shall never decrease `commit_index`; `raft_advance_commit` shall be monotonic, only ever moving `commit_index` forward.
|
||||
6. If `raft_advance_commit(...)` is called while the node's role is not `RAFT_LEADER`, then the Raft Library shall leave `commit_index` unchanged and report a non-success status, mirroring the `raft.tla` `AdvanceCommitIndex` `state[i] = Leader` precondition.
|
||||
7. The Raft Library shall obtain the term of the entry at `Max(agreeIndexes)` exclusively through the existing `on_get_entry_term` callback and shall perform no storage I/O itself.
|
||||
|
||||
### Requirement 2: on_commit_advanced callback triggering the existing commit barrier
|
||||
|
||||
**Objective:** As the XCE controller adapter, I want the library to notify me exactly when the committed position moves forward, so that I can drive the existing synchronous commit barrier / persist-before-ACK durability off the library's safe commit index without the library performing or weakening that durability work.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Raft Library shall define the required callback `void on_commit_advanced(void *ctx, uint64_t commit_index)`, fired with the new `commit_index` after it strictly advances.
|
||||
2. When `raft_advance_commit(...)` advances `commit_index` to a strictly greater value, the Raft Library shall invoke `on_commit_advanced(ctx, commit_index)` exactly once with the new value.
|
||||
3. If `raft_advance_commit(...)` does not change `commit_index` (held by the current-term rule, empty `agreeIndexes`, or non-leader), then the Raft Library shall not invoke `on_commit_advanced`.
|
||||
4. The Raft Library shall treat `on_commit_advanced` as a trigger only: it shall not itself fsync, persist entries, send the client ACK, or otherwise implement the commit barrier — those remain the controller adapter's responsibility (persist-before-ACK durability preserved as policy).
|
||||
5. The Raft Library shall validate `on_commit_advanced` as a required, non-NULL callback at `raft_init`, returning `-EINVAL` when it is NULL, consistent with how existing required callbacks are validated.
|
||||
6. The Raft Library shall not advance `commit_index` or fire `on_commit_advanced` as a side effect of `raft_on_append_entries_reply`; updating per-peer `match_index[]` and advancing the commit index remain separate steps (mirroring `raft.tla`, where `AdvanceCommitIndex` is a distinct action from `HandleAppendEntriesResponse`).
|
||||
|
||||
### Requirement 3: (lastLogTerm, lastLogIndex) vote freshness on the binding vote path (closes G6)
|
||||
|
||||
**Objective:** As any cluster member receiving a `RequestVote`, I want to grant my vote only to a candidate whose log is at least as up-to-date as mine by the `(lastLogTerm, lastLogIndex)` comparison, so that an election can never elect a leader missing a committed entry (Leader Completeness).
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When `raft_on_vote_request(...)` evaluates freshness, the Raft Library shall compute `logOk = (cand_last_log_term > local_last_log_term) ∨ (cand_last_log_term == local_last_log_term ∧ cand_last_log_index ≥ local_last_log_index)`, mirroring the `raft.tla` `HandleRequestVoteRequest` `logOk` predicate (closes G6).
|
||||
2. The Raft Library shall accept the candidate's `last_log_term` alongside the existing `last_log_index` on the vote-request path so the `(lastLogTerm, lastLogIndex)` comparison can be made (extending the eligibility signature or the vote-request payload accordingly).
|
||||
3. The Raft Library shall obtain the voter's own `local_last_log_term` and `local_last_log_index` from its library log model (`on_get_entry_term(log_len)` and `log_len`, with term `0` when the log is empty), consistent with `raft.tla` `LastTerm(log)`.
|
||||
4. If `logOk` is false, then the Raft Library shall deny the vote, even when all other Raft invariants (correct term, no prior vote this term) and application gates would otherwise grant it.
|
||||
5. The Raft Library shall apply the `(lastLogTerm, lastLogIndex)` freshness check without weakening the existing election-safety invariants (stale-term deny, one-vote-per-term, persist-before-grant) or the existing application policy gate (primary-deny / heartbeat stickiness / catch-up), which remain in force.
|
||||
6. The Raft Library shall preserve the existing persist-before-grant ordering: a vote that passes the `(lastLogTerm, lastLogIndex)` freshness check is persisted via `on_persist` before being granted.
|
||||
|
||||
### Requirement 4: (lastLogTerm, lastLogIndex) freshness on the PreVote eligibility path (closes G6 for PreVote)
|
||||
|
||||
**Objective:** As a node answering a PreVote straw poll, I want to apply the same `(lastLogTerm, lastLogIndex)` freshness rule as the binding vote, so that PreVote never green-lights a candidacy that a real election would reject for log staleness — without mutating any persistent state.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When `raft_on_pre_vote_request(...)` evaluates eligibility, the Raft Library shall apply the same `(lastLogTerm, lastLogIndex)` freshness predicate as the binding vote path (Requirement 3.1), using the candidate's advertised `last_log_term` and `last_log_index`.
|
||||
2. The Raft Library shall carry `last_log_term` alongside `last_log_index` on the PreVote eligibility path consistently with the binding vote path, so the two paths apply an identical freshness rule.
|
||||
3. If the `(lastLogTerm, lastLogIndex)` freshness predicate is false, then the Raft Library shall deny the PreVote.
|
||||
4. While answering a PreVote, the Raft Library shall not mutate `currentTerm`, `voted_for`, `voted_for_term`, `role`, or `commit_index`, and shall not invoke `on_persist` — preserving the existing PreVote side-effect-free invariant.
|
||||
5. The Raft Library shall keep the existing PreVote stale-term deny and application eligibility gate (`on_vote_eligible`) in force; the `(lastLogTerm, lastLogIndex)` freshness is applied in addition to them, not in place of them.
|
||||
|
||||
### Requirement 5: Preserve the synchronous commit barrier and durability as policy
|
||||
|
||||
**Objective:** As the operator of an XCE HA cluster, I want the commit-safety upgrade to keep XCE's stronger-than-canonical durability guarantees intact, so that acknowledged work remains durable on a majority and the term-safety fix does not regress availability or durability behavior.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Raft Library shall expose the safe commit index only as a trigger (`on_commit_advanced`); the synchronous commit barrier, persist-before-ACK durability, fsync/group-commit, and the reply gate shall remain the controller adapter's responsibility and shall not be moved into or weakened by the library.
|
||||
2. The Raft Library shall not change the existing PreVote mechanism or priority-leadership behavior; the freshness predicate these paths consult gains `last_log_term`, and the locus of that freshness decision moves from the caller/dispatcher into the library (see Requirement 5.4).
|
||||
3. The Raft Library shall return status codes only and shall never abort a process on any commit-safety or freshness condition, preserving the "controller never self-terminates" invariant.
|
||||
4. The Raft Library shall leave all existing election and log-replication entry points and callbacks (from `raft-library` and `raft-log-replication`) behaviorally unchanged, **except** for: (a) the additive freshness arguments on the vote/PreVote paths, (b) the additive `commit_index` / `on_commit_advanced` state and callback, and (c) the **freshness-decision locus move** — the vote/PreVote entry points gain the `last_log_term` / `last_log_index` arguments **and** the library now owns the `logOk` decision that was previously the caller's/dispatcher's (the binding path no longer delegates freshness to `on_vote_eligible`, and the PreVote path no longer relies on the dispatcher applying freshness before the call).
|
||||
|
||||
### Requirement 6: Deterministic Figure-8 and Leader Completeness verification in the simulator
|
||||
|
||||
**Objective:** As a maintainer, I want the deterministic simulator to drive commit advance and forced leader changes and assert the Figure-8 refusal and Leader Completeness, so that the commit-safety and freshness rules are verifiable without a live cluster.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Raft Simulator shall drive `raft_advance_commit` on the leader after replication so the commit index advances deterministically over the in-memory per-node log backing the existing callbacks.
|
||||
2. The Raft Simulator shall construct a **Figure-8** scenario — a previous-term entry replicated to a quorum but not yet covered by a committed current-term entry — and shall assert that the library does **not** advance `commit_index` over that previous-term entry until an entry from the current term is committed.
|
||||
3. When the current-term entry that covers the previous-term entry is replicated to a quorum and `raft_advance_commit` is driven, the Raft Simulator shall assert that `commit_index` then advances over both entries (indirect commit of the previous-term entry).
|
||||
4. The Raft Simulator shall assert the **Leader Completeness** invariant across forced leader changes: any entry that was committed (present at or below `commit_index` on the committing leader) is present in the log of every node that subsequently becomes leader.
|
||||
5. The Raft Simulator shall exercise the `(lastLogTerm, lastLogIndex)` freshness gate: a candidate with a stale last-log-term shall be denied the vote even when its last-log-index is greater, and a candidate with a fresher last-log-term shall be granted regardless of a smaller last-log-index, asserting the lexicographic comparison.
|
||||
6. The Raft Simulator shall assert that `on_commit_advanced` fires exactly when `commit_index` strictly advances and does not fire when the current-term rule holds the commit index, so the trigger semantics consumed by the controller barrier are verified.
|
||||
23
.kiro/specs/raft-commit-safety/spec.json
Normal file
23
.kiro/specs/raft-commit-safety/spec.json
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"feature_name": "raft-commit-safety",
|
||||
"created_at": "2026-06-21T12:00:00Z",
|
||||
"updated_at": "2026-06-21T12:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true,
|
||||
"note": "Auto-approved (-y mode). Adds the commit-safety + election-freshness half of raft.tla on top of raft-log-replication's library log. raft_advance_commit (AdvanceCommitIndex, raft.tla:259) advances commit_index to Max(agreeIndexes from match_index[]) ONLY when on_get_entry_term(Max)==currentTerm (Figure-8 current-term rule, G4); new callback on_commit_advanced(ctx, commit_index) TRIGGERS the existing synchronous commit barrier / persist-before-ACK durability (does NOT replace it). Vote/PreVote freshness upgraded from LSN-only to (lastLogTerm,lastLogIndex) lexicographic compare (HandleRequestVoteRequest logOk, raft.tla:285, G6) carried via last_log_term in raft_on_vote_request + on_vote_eligible/vote-request payload. Simulator asserts Figure-8 refusal + Leader Completeness. Boundary: owns commit-index advance + vote freshness in src/lib/raft ONLY; NOT the log model/AppendEntries (raft-log-replication), the barrier/durability mechanism (controller), config markers/snapshot (ha-term-safety), or the conformance harness (raft-sim-conformance). Depends: raft-log-replication (HARD). Closes bug-851 G4/G6."
|
||||
}
|
||||
165
.kiro/specs/raft-commit-safety/tasks.md
Normal file
165
.kiro/specs/raft-commit-safety/tasks.md
Normal file
|
|
@ -0,0 +1,165 @@
|
|||
# Implementation Plan
|
||||
|
||||
> **TDD-mandatory** (`testing.md`): every implementation sub-task is preceded by a RED test in `src/lib/testsuite/test_raft.c` (and the simulator backing in `raft_sim.c`) that fails before the `raft.c` change and passes after. Build/run library tests via `/xce-build lib`.
|
||||
>
|
||||
> **Spec boundary** (applies to ALL tasks): this spec owns the commit-index advance (`raft.tla` `AdvanceCommitIndex` + the current-term Figure-8 rule, closes G4) and the `(lastLogTerm, lastLogIndex)` vote/PreVote freshness (`raft.tla` `HandleRequestVoteRequest` `logOk`, closes G6) **in `src/lib/raft` only**. It does **NOT** own the log model / AppendEntries / per-peer `match_index[]` (those come from `raft-log-replication` and are reused verbatim), the synchronous commit-barrier / durability mechanism (controller — only TRIGGERED via `on_commit_advanced`), config markers / upgrade snapshot (`ha-term-safety`), or the conformance harness (`raft-sim-conformance`). No task here edits any controller file; the controller adapter is described, not built.
|
||||
>
|
||||
> **Depends (HARD): `raft-log-replication`** — needs `log_len`, `match_index[]`, and `on_get_entry_term` already present in `raft.{c,h}` and the in-memory per-node log + AppendEntries routing already in `raft_sim.c`.
|
||||
|
||||
## 1. Header contract: commit_index, on_commit_advanced, raft_advance_commit, freshness signatures
|
||||
|
||||
- [ ] 1.1 Extend `raft.h` with the commit-index state and restore field
|
||||
- Add `uint64_t commit_index;` to `raft_state_t`
|
||||
- Add `uint64_t initial_commit_index;` to `raft_config_t` (restored committed LSN at boot, default 0; library never reads storage — consistent with `initial_term`/`initial_log_len`)
|
||||
- Observable completion: `raft.h` compiles; `sizeof(raft_state_t)` grows by the new field; no entry-byte storage added
|
||||
- _Requirements: 1.1, 1.5_
|
||||
- _Boundary: raft.h_
|
||||
|
||||
- [ ] 1.2 Add the `on_commit_advanced` REQUIRED callback to `raft_callbacks_t` (verbatim contract)
|
||||
- Declare `void on_commit_advanced(void *ctx, uint64_t commit_index)` with the exact signature from design.md / brief
|
||||
- Document it as REQUIRED, fired only on strict advance, and as a TRIGGER for the existing commit barrier (never performs durability itself)
|
||||
- Observable completion: the function-pointer member exists in `raft_callbacks_t` with the verbatim signature; header doc comment matches the Authoritative shared contract
|
||||
- _Requirements: 2.1, 2.4, 5.1_
|
||||
- _Boundary: raft.h_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 1.3 Declare `raft_advance_commit` and extend the vote/PreVote signatures with last_log_term/last_log_index
|
||||
- Declare `int raft_advance_commit(raft_state_t *s)`
|
||||
- Extend `raft_on_vote_request` and `raft_on_pre_vote_request` to accept `cand_last_log_term` and `cand_last_log_index` (the chosen "library owns logOk" plumbing — Option B in design.md)
|
||||
- Update the header doc comments to describe the new logOk gate and the commit-advance action
|
||||
- Observable completion: header declares `raft_advance_commit` and the extended vote/PreVote signatures; a stub TU including `raft.h` compiles
|
||||
- _Requirements: 1.2, 2.2, 3.2, 4.2_
|
||||
- _Boundary: raft.h_
|
||||
- _Depends: 1.2_
|
||||
|
||||
## 2. Init validation and commit-index restore
|
||||
|
||||
- [ ] 2.1 RED+GREEN: `raft_init` requires `on_commit_advanced` and restores `initial_commit_index`
|
||||
- Test: `raft_init` returns `-EINVAL` when `on_commit_advanced` is NULL; returns 0 when present; `s->commit_index == cfg->initial_commit_index` after init
|
||||
- Implement the NULL check in `raft_init` (mirroring the existing required-callback checks); restore `s->commit_index = cfg->initial_commit_index`
|
||||
- Observable completion: new `-EINVAL` test case passes; restore assertion passes; existing election + log-replication init tests still green
|
||||
- _Requirements: 2.5, 1.1_
|
||||
- _Boundary: raft.c, test_raft.c_
|
||||
- _Depends: 1.3_
|
||||
|
||||
## 3. AdvanceCommitIndex with the current-term Figure-8 rule (raft.tla:259/270, closes G4)
|
||||
|
||||
- [ ] 3.1 RED: commit-advance tests (current-term advance, Figure-8 hold, monotonic, non-leader)
|
||||
- Test (advance): leader with `match_index[]` quorum-agreeing through index `k`, entry at `k` from `currentTerm` → `commit_index` advances to `k`, `on_commit_advanced(k)` fired once
|
||||
- Test (Figure-8 hold): quorum agrees through a previous-term entry at `k` with no current-term entry above → `commit_index` unchanged, `on_commit_advanced` NOT fired
|
||||
- Test (empty agreeIndexes): no index reaches quorum → unchanged, no callback
|
||||
- Test (monotonic): a later call computing a smaller `Max(agreeIndexes)` never lowers `commit_index`
|
||||
- Test (non-leader): `raft_advance_commit` returns `-EPERM`, no change, no callback
|
||||
- Observable completion: tests compile and FAIL (no implementation yet)
|
||||
- _Requirements: 1.2, 1.3, 1.4, 1.5, 1.6, 2.2, 2.3_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [ ] 3.2 GREEN: implement `raft_advance_commit`
|
||||
- Leader-only guard (`-EPERM` otherwise); compute `Max(agreeIndexes)` by scanning candidate indexes from `log_len` downward, seeding the agree-count with self, until a quorum `((n_peers/2)+1)` is reached
|
||||
- Apply the current-term rule: advance to `Max(agreeIndexes)` only when `on_get_entry_term(Max) == s->term`; else hold
|
||||
- Monotonic: only ever move `commit_index` forward; fire `on_commit_advanced(commit_index)` exactly once on a strict advance; read entry terms only via `on_get_entry_term`
|
||||
- Observable completion: 3.1 tests pass; the current-term check is unmistakably present before the advance in `raft.c`; `commit_index <= log_len` holds
|
||||
- _Requirements: 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.2, 2.3_
|
||||
- _Boundary: raft.c_
|
||||
- _Depends: 3.1_
|
||||
|
||||
- [ ] 3.3 RED+GREEN: commit-advance is a separate action from append-entries reply
|
||||
- Test: a `raft_on_append_entries_reply` that updates `match_index[]` to a committable quorum does NOT by itself change `commit_index` or fire `on_commit_advanced`; only a subsequent `raft_advance_commit` does
|
||||
- Verify (no implementation change expected in `raft_on_append_entries_reply` — it is owned upstream and must stay untouched); if a regression is found, the fix belongs upstream, not here
|
||||
- Observable completion: test passes; `raft_on_append_entries_reply` remains free of any commit-advance side effect
|
||||
- _Requirements: 2.6_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 3.2_
|
||||
|
||||
## 4. (lastLogTerm, lastLogIndex) vote freshness (raft.tla:285, closes G6)
|
||||
|
||||
- [ ] 4.1 RED: binding-vote freshness tests
|
||||
- Test: candidate with stale `last_log_term` denied even when `last_log_index` is greater; candidate with fresher `last_log_term` granted with a smaller `last_log_index`; tie on term grants only when `cand_last_log_index >= log_len`; the voter's own last-log term/index come from `on_get_entry_term(log_len)`/`log_len` (term 0 when empty)
|
||||
- Test: `!logOk` denies even when `on_vote_eligible` would grant (logOk applied BEFORE the application gate); persist-before-grant unchanged on the grant path
|
||||
- Observable completion: tests compile and FAIL
|
||||
- _Requirements: 3.1, 3.3, 3.4, 3.5, 3.6_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 1.3_
|
||||
|
||||
- [ ] 4.2 GREEN: implement the logOk gate in `raft_on_vote_request`
|
||||
- Compute `local_last_log_term = (log_len > 0) ? on_get_entry_term(log_len) : 0`; `logOk = (cand_last_log_term > local_last_log_term) || (cand_last_log_term == local_last_log_term && cand_last_log_index >= log_len)`
|
||||
- Insert the logOk deny BEFORE `on_vote_eligible`, after the existing stale-term/UpdateTerm/one-vote-per-term steps; leave persist-before-grant and all existing invariants in place
|
||||
- Observable completion: 4.1 tests pass; no regression in existing vote tests; the logOk block is clearly before `on_vote_eligible` in `raft.c`
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_
|
||||
- _Boundary: raft.c_
|
||||
- _Depends: 4.1_
|
||||
|
||||
## 5. (lastLogTerm, lastLogIndex) PreVote freshness (raft.tla:285, side-effect-free)
|
||||
|
||||
- [ ] 5.1 RED: PreVote freshness tests (identical rule, no side effects)
|
||||
- Test: `raft_on_pre_vote_request` applies the SAME `(lastLogTerm, lastLogIndex)` logOk as the binding vote; a stale tuple is denied; the existing stale-term deny and `on_vote_eligible` gate remain in force
|
||||
- Test: across grant and deny, no mutation of `term`/`voted_for`/`voted_for_term`/`role`/`commit_index` and no `on_persist` call (PreVote invariant)
|
||||
- Observable completion: tests compile and FAIL
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 1.3_
|
||||
|
||||
- [ ] 5.2 GREEN: implement the logOk gate in `raft_on_pre_vote_request`
|
||||
- Apply the identical logOk predicate (factored to match 4.2) before/with the existing eligibility gate; mutate nothing; no `on_persist`
|
||||
- Observable completion: 5.1 tests pass; no regression in existing PreVote tests; the predicate matches the binding-vote path exactly
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
|
||||
- _Boundary: raft.c_
|
||||
- _Depends: 5.1_
|
||||
|
||||
## 6. Simulator backing: commit-advance driver + freshness plumbing
|
||||
|
||||
- [ ] 6.1 Wire `on_commit_advanced` and a commit-advance driver in `raft_sim.c`
|
||||
- Add a per-`raft_sim_node_t` `commit_index` mirror updated from `on_commit_advanced`; implement the callback (record the value + count fires for assertions)
|
||||
- Add a sim helper to drive `raft_advance_commit` on a given node (or all leaders) after delivery
|
||||
- Pass each node's restored `initial_commit_index` (default 0) in `raft_sim_init`; wire `on_commit_advanced` for every node
|
||||
- Observable completion: after replication + delivery, driving the helper advances the leader's mirrored `commit_index`; the callback fire-count is observable
|
||||
- _Requirements: 6.1, 6.6, 2.5_
|
||||
- _Boundary: raft_sim.c_
|
||||
- _Depends: 3.2_
|
||||
|
||||
- [ ] 6.2 Plumb `(last_log_term, last_log_index)` through the sim vote/PreVote routing
|
||||
- Update the sim's vote and PreVote delivery to read the candidate's last-log `(term, index)` from its in-memory log (`term at log_len` / `log_len`) and pass them to the extended `raft_on_vote_request` / `raft_on_pre_vote_request`
|
||||
- Observable completion: existing election sim tests still pass with the extended signatures; a node with a stale last-log term is denied a vote it would previously have won on LSN alone
|
||||
- _Requirements: 6.5, 3.2, 4.2_
|
||||
- _Boundary: raft_sim.c_
|
||||
- _Depends: 4.2, 5.2_
|
||||
|
||||
## 7. Simulator invariant tests (Figure-8, Leader Completeness, freshness, trigger semantics)
|
||||
|
||||
- [ ] 7.1 Figure-8 commit-refusal test (G4)
|
||||
- Test: construct a previous-term entry replicated to a quorum but not covered by a committed current-term entry; drive `raft_advance_commit` and assert `commit_index` does NOT move over it; then replicate a current-term entry above it to a quorum, drive `raft_advance_commit`, and assert `commit_index` advances over BOTH (indirect commit)
|
||||
- Observable completion: test passes end-to-end via the sim; the held phase and the released phase are both asserted
|
||||
- _Requirements: 6.2, 6.3_
|
||||
- _Boundary: test_raft.c, raft_sim.c_
|
||||
- _Depends: 6.1_
|
||||
|
||||
- [ ] 7.2 Leader Completeness across forced leader changes
|
||||
- Test: drive replication to a commit, then force leader changes (timeout / partition); assert any entry committed (≤ `commit_index` on the committing leader) is present in the log of every node that subsequently becomes leader
|
||||
- Observable completion: test passes; no future leader is missing a committed entry across the churn
|
||||
- _Requirements: 6.4_
|
||||
- _Boundary: test_raft.c, raft_sim.c_
|
||||
- _Depends: 6.2, 7.1_
|
||||
|
||||
- [ ] 7.3 Freshness-gate lexicographic exercise
|
||||
- Test: drive votes with mismatched `(last_log_term, last_log_index)` tuples and assert the lexicographic outcome — stale-term-but-higher-index denied; fresher-term-but-lower-index granted; tie-term decided by index
|
||||
- Observable completion: test passes for both vote and PreVote paths
|
||||
- _Requirements: 6.5_
|
||||
- _Boundary: test_raft.c, raft_sim.c_
|
||||
- _Depends: 6.2_
|
||||
|
||||
- [ ] 7.4 Trigger-semantics test for `on_commit_advanced`
|
||||
- Test: `on_commit_advanced` fires exactly once on a strict advance and does NOT fire while the current-term rule holds the commit index (Figure-8 held phase) or when not leader
|
||||
- Observable completion: test passes; the callback fire-count matches the number of strict advances
|
||||
- _Requirements: 6.6, 2.2, 2.3_
|
||||
- _Boundary: test_raft.c, raft_sim.c_
|
||||
- _Depends: 6.1, 7.1_
|
||||
|
||||
## 8. Verification
|
||||
|
||||
- [ ] 8.1 Full library suite green under ASAN
|
||||
- Run `/xce-build lib`; all existing election + log-replication tests plus the new commit-safety and freshness tests pass under ASAN with zero leaks
|
||||
- Observable completion: test binary exits 0; ASAN clean; no regression in the election-core or log-replication tests
|
||||
- _Requirements: 1.2, 1.3, 2.2, 3.1, 4.1, 6.2, 6.4_
|
||||
- _Boundary: src/lib/raft, src/lib/testsuite_
|
||||
- _Depends: 7.1, 7.2, 7.3, 7.4_
|
||||
|
|
@ -179,3 +179,118 @@ Overall remaining work: **S–M total, Low–Medium risk** — the library and i
|
|||
- **R3**: Confirm `raft_check_and_start_election` ignoring the `raft_start_election` return code
|
||||
(raft.c:287–289) is safe in all adapter paths (comment claims -EALREADY unreachable; verify when an
|
||||
inbound vote request flips role between RDHUP and the delayed check).
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
Post-implementation audit. Spec phase = implemented, tasks 24/24 [x]. This pass re-verifies every
|
||||
EARS requirement against the working tree and reconciles the prior audit's open items (C2, R1–R3).
|
||||
Assets are tagged: **[CLOSED]** verified met in code, **[CLOSED-AS-BUILT]** met but behavior differs
|
||||
from literal req text (improvement), **[RESIDUAL]** doc/hygiene gap, **[SCOPE-DRIFT]** code beyond spec.
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
| Req | Acceptance | Asset (file:line) | Status |
|
||||
|-----|-----------|-------------------|--------|
|
||||
| 1.1 | per-node state fields | `raft.h:28-50` raft_state_t (term/role/voted_for/voted_for_term/n_peers) | CLOSED |
|
||||
| 1.2 | init term=0 role=FOLLOWER voted_for=-1 | `raft.c:40-56` raft_init; defaults via cfg {0,-1} | CLOSED |
|
||||
| 1.3 | no transport types in header | `raft.h:4-6` includes only errno/stdint/string; no endpoint_t/SSL_CTX/cluster_t | CLOSED |
|
||||
| 2.1 | election: incr term + self-vote + send RPCs | `raft.c:68-71,98-102` | CLOSED |
|
||||
| 2.2 | single-node → immediate LEADER | `raft.c:87-92` returns RAFT_ELECTED_IMMEDIATELY | CLOSED |
|
||||
| 2.3 | already CANDIDATE/LEADER → error, no state change | `raft.c:64-65` -EALREADY | CLOSED |
|
||||
| 3.1 | term>cur → update,clear vote,FOLLOWER,evaluate | `raft.c:125-137` (also steps down leader via on_become_follower) | CLOSED-AS-BUILT |
|
||||
| 3.2 | term<cur → deny + return cur term | `raft.c:114-119` | CLOSED |
|
||||
| 3.3 | same term, diff voted_for → deny | `raft.c:161-168` (gated on voted_for_term==term) | CLOSED |
|
||||
| 3.4 | freshness gate pass → grant + persist | `raft.c:170-190`; freshness split: tag/lsn floor enforced caller-side (cluster.c:363), COMMITTING/liveness in on_vote_eligible (`ha.c:299` raft_cb_vote_eligible) | CLOSED-AS-BUILT |
|
||||
| 3.5 | LEADER/COMMITTING → deny all | `raft.c:148-152` LEADER deny; COMMITTING enum removed (ha-single-leader-role 93dfc02), now an on_vote_eligible policy | CLOSED-AS-BUILT |
|
||||
| 4.1 | quorum ceil((n+1)/2) → LEADER once | `raft.c:372-383`, quorum=(n_peers/2)+1 == ceil for odd n | CLOSED |
|
||||
| 4.2 | reply term>cur → FOLLOWER + on_become_follower | `raft.c:346-355` | CLOSED |
|
||||
| 4.3 | post-decision no-op | `raft.c:358-362` election_decided guard → RAFT_PENDING | CLOSED |
|
||||
| 4.4 | all replied, no quorum → FOLLOWER | `raft.c:386-393` replies_received==n_peers-1 | CLOSED |
|
||||
| 5.1 | leader-lost → clear vote + delay(priority*timeout) | `raft.c:417-425` | CLOSED |
|
||||
| 5.2 | voted non-self during delay → skip | `raft.c:429-433` RAFT_SKIP_ELECTION | CLOSED |
|
||||
| 5.3 | not voted / self → start election | `raft.c:435-438` RAFT_ELECTION_STARTED | CLOSED |
|
||||
| 6.1 | start election → on_schedule_timeout | `raft.c:105` | CLOSED |
|
||||
| 6.2 | timeout undecided → on_become_follower + cancel | `raft.c:409-414` | CLOSED |
|
||||
| 6.3 | timeout decided → on_cancel_timeout, no-op | `raft.c:401-407` | CLOSED |
|
||||
| 7.1 | raft_callbacks_t with required fn ptrs | `raft.h:79-117` (9 fields incl. on_send_pre_vote_request) | CLOSED |
|
||||
| 7.2 | void* ctx passed unchanged | `raft.h` ctx threaded; `raft.c` passes s->ctx everywhere | CLOSED |
|
||||
| 7.3 | NULL required callback → non-zero | `raft.c:23-38` 7 required-cb checks → -EINVAL | CLOSED |
|
||||
| 8.1 | deterministic N-node simulator | `raft_sim.c`, `raft.h:264-342` (XCE_TESTING) | CLOSED |
|
||||
| 8.2 | 4 invariant cases (a-d) | `test_raft.c`: A2 one-leader, B1/B4 deny-second, A1 single-node, E*/sim timeout | CLOSED |
|
||||
| 8.3 | compiles + tests pass under ASAN (leaks=0) | meson `raft/raft.c` (l.109) + `test_raft.c` (l.179); 209 RUN_TEST/test fns | CLOSED (re-run, see V1) |
|
||||
| 9.1 | adapter implements all callbacks, no dup state | `ha.c:1309-1320` s_real_cbs wires all 10; production state in raft_state_t | CLOSED |
|
||||
| 9.2 | ha delegates term/voted_for/quorum to library | `ha.c:4382-4522` ha_start_election_async→raft_start_election; `cluster.c:418` vote req; `ha.c:3237` vote reply | CLOSED-AS-BUILT |
|
||||
| 9.3 | XCE_TESTING hooks still functional | `ha_rdhup_election_hook` (ha.c:4528), ha_phase2 seams retained | CLOSED |
|
||||
|
||||
**Score: 24/24 acceptance-criteria groups CLOSED.** No requirement is unimplemented. 5 are CLOSED-AS-BUILT
|
||||
(real code is a strict superset/improvement over literal req text — see residual R-DOC).
|
||||
|
||||
### Reconciliation of Prior Audit Open Items
|
||||
|
||||
- **C2 (voted_for_term restore double-vote) — NOW FIXED.** `raft.c:47` sets
|
||||
`voted_for_term = (initial_voted_for != -1) ? initial_term : 0` at init, closing the post-restart
|
||||
second-vote window. Regression test `test_proto_H6_restored_vote_blocks_second_vote` (test_raft.c:2100,
|
||||
registered l.3978) guards it. Prior research's "only candidate correctness gap" is resolved (commit 9821ec4).
|
||||
- **R3 / Req 9.2 dual vote path — DOWNGRADED.** Legacy `handle_request_vote` (ha.c:2608) is now
|
||||
`#ifdef XCE_TESTING`-only — the production vote path is exclusively `cluster.c:418 → raft_on_vote_request`.
|
||||
The duplicated election-state logic no longer runs in production; it remains only as a test fixture for
|
||||
`test_ha_vote_gate.c` / `test_ha_election.c`. No production split-brain risk from the dual path. Residual:
|
||||
the test-only copy can drift from raft.c semantics over time (R-DUP below).
|
||||
- **R1 (even-n_peers policy) — STILL OPEN.** `raft_init` rejects even n_peers>1 with `-EINVAL`
|
||||
(raft.c:21-22), test `test_raft_init_even_peers_returns_einval`. Not stated in requirements; a 2-node HA
|
||||
deploy would silently fail raft_init. Needs requirements note + startup-time config validation.
|
||||
|
||||
### Residual Gaps (non-blocking)
|
||||
|
||||
- **R-DOC (Low):** requirements.md 3.4/3.5 still describe COMMITTING as an in-library role gate and the
|
||||
freshness gate as in-library; as-built, COMMITTING enum is removed and freshness is split caller-side +
|
||||
on_vote_eligible. Requirements text lags the (better) implementation. Update for traceability.
|
||||
- **R-DUP (Low):** test-only `handle_request_vote` duplicates vote-gate logic; risk of semantic drift from
|
||||
raft.c. Consider retargeting `test_ha_vote_gate.c` onto `raft_on_vote_request` directly.
|
||||
- **R1 (Medium):** even-n_peers rejection undocumented; surfaces as silent raft_init -EINVAL rather than a
|
||||
clear startup error. Add config-load validation OR a requirement clause.
|
||||
- **R-SCOPE / SCOPE-DRIFT (informational, not a gap):** PreVote (raft_begin_pre_vote,
|
||||
raft_on_pre_vote_request, raft_on_pre_vote_reply, on_send_pre_vote_request, RAFT_PRE_VOTE_* codes;
|
||||
raft.c:193-336, raft.h:160-259) lives in the raft-library files but belongs to the separate **ha-prevote**
|
||||
spec (commits 449b8e9–e6b234a). It is purely additive, mutates no persistent Raft state, and does not
|
||||
violate any raft-library requirement. Flag only so the raft-library boundary doc reflects the co-tenancy:
|
||||
raft.h/raft.c now host two specs' surfaces.
|
||||
|
||||
### Integration Risks
|
||||
|
||||
- **IR-1 (Low):** PreVote tally arrays are bounded by `RAFT_MAX_PEERS=8` (raft.h:18); `raft_on_pre_vote_reply`
|
||||
range-guards (raft.c:287). Real-vote path has no equivalent fixed array (uses scalar counters) so it is
|
||||
unbounded-safe. Mismatch is intentional but worth a one-line note in design.
|
||||
- **IR-2 (Low):** `raft_check_and_start_election` discards `raft_start_election` return (raft.c:436-437);
|
||||
comment claims -EALREADY unreachable because raft_on_leader_lost forces FOLLOWER. Valid for the current
|
||||
adapter, but an inbound vote/AppendEntries that flips role between RDHUP and the delayed fire could make it
|
||||
reachable. Low likelihood; assert instead of (void) for defense-in-depth.
|
||||
|
||||
### Open Work — Option Analysis
|
||||
|
||||
**Option A — Doc-only closure (recommended).** Update requirements.md 3.4/3.5/9.2 to as-built; add R1
|
||||
even-n_peers clause; note PreVote co-tenancy + IR-1/IR-2 in design.md. Then `/kiro-verify-completion`.
|
||||
- Effort: **S** (~1-2h, docs only). Risk: **Low**. No code change, no regression surface.
|
||||
|
||||
**Option B — Doc closure + hygiene.** Option A plus: (1) startup config validation for even n_peers with a
|
||||
loud error, (2) retarget test-only `handle_request_vote` onto raft_on_vote_request (kill R-DUP), (3) convert
|
||||
IR-2 `(void)rc` to an assert.
|
||||
- Effort: **M** (~half day). Risk: **Low-Medium** (touches ha.c test seam + config load; rerun full
|
||||
controller + lib test suites).
|
||||
|
||||
**Option C — Defer, accept as-is.** Ship implemented; track R-DOC/R-DUP/R1 as backlog. The only real
|
||||
correctness item (C2) is already fixed and tested, so functional risk of deferring is minimal.
|
||||
- Effort: **none**. Risk: **Low** functionally; **Medium** for spec/code traceability drift over time.
|
||||
|
||||
### Validation To Re-Run
|
||||
|
||||
- **V1:** `/xce-build lib` and `/xce-build controller` with `ASAN_OPTIONS=detect_leaks=0` — no build dir
|
||||
present in the worktree at audit time; the test_raft.c suite (209 cases) and controller tests should be
|
||||
re-run to confirm green on the current tree (which now includes the PreVote additions + C2 fix, post-dating
|
||||
the last recorded task-7 ASAN run).
|
||||
|
||||
**Bottom line:** raft-library is functionally complete — 24/24 met, the sole prior correctness gap (C2) is
|
||||
fixed and regression-tested, and the production dual-vote-path risk is eliminated (legacy path is test-only).
|
||||
Remaining work is documentation/hygiene (Option A, S effort, Low risk) plus a clean ASAN re-run (V1).
|
||||
|
|
|
|||
41
.kiro/specs/raft-log-replication/brief.md
Normal file
41
.kiro/specs/raft-log-replication/brief.md
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
# Brief: raft-log-replication
|
||||
|
||||
## Problem
|
||||
`src/lib/raft/raft.{c,h}` is an **election-only** Raft library (terms, votes, leader election, PreVote) with a clean callback interface and a deterministic simulator (`raft_sim.c`). It has **no log, no AppendEntries, no commit**. The replicated log is hand-rolled in the controller (`persist.c` WAL, `repl.c` catch-up streaming, `ha.c`, `ha_convergence.c`) and **violates the Raft log invariants of `raft/raft.tla`**, which root-caused **bug-851** (silent committed-prefix divergence under chaotic failover). Authoritative gap analysis: `.kiro/specs/ha-term-safety/TLA-gap-analysis.md`.
|
||||
|
||||
This spec closes the four **log-half** gaps by making the library own the log, strictly following `raft.tla`:
|
||||
- **G1** — per-entry term is dead (WAL `hdr.term` always 0; `p->term` never set from `ha->term`).
|
||||
- **G2** — replication matches LSN contiguity only; no `prevLogTerm` check (TLA+ `AppendEntries`/`HandleAppendEntriesRequest` `logOk`).
|
||||
- **G3** — no `(term,lsn)` conflict truncation (TLA+ conflict branch: delete the divergent entry + all after).
|
||||
- **G5** — a node adopts a higher term only on **vote** RPCs; the AppendEntries/heartbeat path never advances `currentTerm` (TLA+ `UpdateTerm` fires on **any** higher-term RPC). Confirmed live: a rejoined standby reported `term:2` under a term-3 primary → term reuse → two leaders per term.
|
||||
|
||||
## Approach
|
||||
Extend the library to own the replicated log, **strictly mirroring the `raft.tla` actions**, keeping the core **pure / synchronous / deterministic** (no I/O, no threads) so it stays simulator-verifiable. All side effects go through new callbacks the controller adapter implements over the existing WAL/repl transport.
|
||||
|
||||
### TLA+ actions implemented (raft.tla line refs)
|
||||
- `ClientRequest(i,v)` (246) → `raft_log_append(...)`: append `[term ↦ currentTerm, value]`.
|
||||
- `AppendEntries(i,j)` (204) → emit via `on_send_append_entries` with `prevLogIndex`, `prevLogTerm`, ≤1+ entries, `leaderCommit`. Per-peer `nextIndex`/`matchIndex` (TLA+ leaderVars).
|
||||
- `HandleAppendEntriesRequest(i,j,m)` (327) → `raft_on_append_entries_request(...)`: `logOk` = `prevLogIndex==0 ∨ log[prevLogIndex].term==prevLogTerm`; reject (stale term / `¬logOk`), candidate→follower on equal term, accept (already-have / conflict-truncate-one / append).
|
||||
- `HandleAppendEntriesResponse(i,j,m)` (393) → `raft_on_append_entries_reply(...)`: success → advance `nextIndex`/`matchIndex`; fail → decrement `nextIndex`.
|
||||
- `UpdateTerm(i,j,m)` (406) → applied on **every** incoming RPC including AppendEntries/heartbeat (closes G5): higher term ⇒ adopt term, clear vote, step down to follower.
|
||||
|
||||
### Authoritative shared contract (downstream specs MUST reference this verbatim)
|
||||
Log index model: the library log index is 1-based (TLA+) and maps 1:1 to the controller WAL **LSN**. Per-entry `term` is `uint64_t`. New REQUIRED callbacks (pure; controller implements I/O):
|
||||
- `int on_persist_entry(void *ctx, uint64_t index, uint64_t term, const void *entry, size_t len)` — durably append/overwrite at index.
|
||||
- `uint64_t on_get_entry_term(void *ctx, uint64_t index)` — term of local entry at index (0 if absent).
|
||||
- `int on_send_append_entries(void *ctx, int peer_idx, uint64_t prev_log_index, uint64_t prev_log_term, uint64_t leader_commit, /* entries handle */ ...)` — transmit.
|
||||
- `int on_truncate_from(void *ctx, uint64_t index)` — delete local entries at `index..end`.
|
||||
New library entry points: `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`. Existing election callbacks/behaviour unchanged.
|
||||
|
||||
## Scope
|
||||
- **In**: library log model with per-entry term; `ClientRequest` append; AppendEntries send (per-peer next/match index); `HandleAppendEntriesRequest` accept/reject/conflict-truncate/append by `(term,index)`; `UpdateTerm` on every RPC; controller adapter that maps callbacks onto the existing WAL/repl transport (WAL `hdr.term` stamped from `ha->term`, carried in the repl envelope, read back on recovery); simulator coverage asserting **Log Matching**.
|
||||
- **Out**: `AdvanceCommitIndex` / commit-index safety + `(term,lsn)` vote freshness → `raft-commit-safety`. The full conformance harness → `raft-sim-conformance`. Config-change markers + upgrade snapshot → `ha-term-safety`. Async I/O, fan-out scheduling, the sync commit barrier, snapshot transport → controller callback adapters (not the pure core).
|
||||
|
||||
## Preserve as policy (do NOT canonicalize away)
|
||||
PreVote, priority leadership, persist-before-ACK durability + synchronous commit barrier (consumed in `raft-commit-safety`), "controller never self-terminates" (library returns statuses; controller owns liveness/fencing).
|
||||
|
||||
## Verification
|
||||
Extend `raft_sim.c`: drive AppendEntries across partitions, message drop/dup/reorder, and restarts; assert the **Log Matching** invariant (same `(index,term)` ⇒ identical prefix) and that a follower truncates a divergent suffix. Replace the controller's LSN-only reconcile with `(term,index)` matching delegated to the library.
|
||||
|
||||
## Dependencies
|
||||
`raft-library` (election core, implemented `[x]`). Feeds `raft-commit-safety`, `raft-sim-conformance`, and `ha-term-safety` integration. Closes bug-851 G1/G2/G3/G5.
|
||||
411
.kiro/specs/raft-log-replication/design.md
Normal file
411
.kiro/specs/raft-log-replication/design.md
Normal file
|
|
@ -0,0 +1,411 @@
|
|||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose**: This feature extends the election-only Raft library (`src/lib/raft/raft.{c,h}`) into a strict implementation of the **log half** of `raft/raft.tla` (Ongaro canonical), giving the library ownership of the replicated log. It closes the four log-half gaps of bug-851 (G1/G2/G3/G5): per-entry term, `prevLogTerm` log-matching, `(term, index)` conflict truncation, and term adoption on the AppendEntries/heartbeat path.
|
||||
|
||||
**Users**: The XCE controller (as a Raft node), the deterministic simulator (`raft_sim.c`), and three downstream specs that consume this seam verbatim — `raft-commit-safety`, `raft-sim-conformance`, and `ha-term-safety`.
|
||||
|
||||
**Impact**: Today the replicated log is hand-rolled across the controller (`persist.c`/`repl.c`/`ha.c`/`ha_convergence.c`) and matches by LSN contiguity only. After this feature, the library owns log growth, AppendEntries emission/handling, per-peer progress, and `(term, index)` truncation as pure, synchronous, deterministic state transitions; the controller becomes a thin adapter that performs I/O behind four new callbacks. The on-disk WAL format is unchanged (the term is a value fix, not a layout change).
|
||||
|
||||
### Goals
|
||||
- Add a library-owned log model where every entry carries a `uint64_t` term, index 1-based mapping 1:1 to the WAL LSN (G1).
|
||||
- Implement `ClientRequest`, `AppendEntries` emission, `HandleAppendEntriesRequest`, and `HandleAppendEntriesResponse` exactly as `raft.tla` defines them (G2/G3).
|
||||
- Apply `UpdateTerm` on **every** RPC including AppendEntries/heartbeat (G5).
|
||||
- Define the four new callbacks **verbatim** per the brief's Authoritative shared contract.
|
||||
- Extend `raft_sim.c` to assert the **Log Matching** invariant and divergent-suffix truncation under partitions, drop/dup/reorder, and restart.
|
||||
|
||||
### Non-Goals
|
||||
- **No commit-index advance** (`AdvanceCommitIndex`, G4) and **no Figure-8 current-term commit rule** — owned by `raft-commit-safety`.
|
||||
- **No `(lastLogTerm, lastLogIndex)` vote freshness** (G6) — owned by `raft-commit-safety`.
|
||||
- **No config-change markers / upgrade snapshot** — owned by `ha-term-safety`.
|
||||
- **No async I/O, fan-out scheduling, commit barrier, or snapshot transport** — these live in controller-side callback adapters, not the pure core.
|
||||
- **No heavy controller WAL/repl/recovery wiring** — this spec defines only the seam contract and a thin adapter description; the integration is `ha-term-safety`.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
- The library log model: per-entry `uint64_t` term, 1-based index, the in-library leader progress vectors (`next_index[]`, `match_index[]`).
|
||||
- New entry points: `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`.
|
||||
- The four new callbacks in `raft_callbacks_t`: `on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`.
|
||||
- `UpdateTerm`-on-any-RPC applied on the AppendEntries/heartbeat path.
|
||||
- The `(term, index)` reconcile seam: the library decides match / truncate / append; the controller routes the decision.
|
||||
- Simulator extensions in `raft_sim.c` + tests in `src/lib/testsuite/test_raft.c` asserting Log Matching.
|
||||
|
||||
### Out of Boundary
|
||||
- Commit-index advance and current-term commit safety (`raft-commit-safety`, G4).
|
||||
- Vote-freshness `(term, index)` comparison in `RequestVote`/PreVote (`raft-commit-safety`, G6).
|
||||
- Configuration markers in the unified log + the change-boundary snapshot (`ha-term-safety`).
|
||||
- The controller's WAL byte layout, repl envelope serialization, recovery loop, commit barrier, fsync/durability, and snapshot transport internals (`ha-term-safety` adapter). This spec specifies only **what the adapter must route**, not its internal mechanics.
|
||||
|
||||
### Allowed Dependencies
|
||||
- The implemented election core (`raft-library`): `raft_state_t`, `raft_callbacks_t`, `raft_init`, the vote/PreVote path. All unchanged.
|
||||
- `errno.h`, `stdint.h`, `string.h` only (the library has no other dependencies and must stay free-standing for the simulator).
|
||||
- The simulator harness (`raft_sim.c`) for verification.
|
||||
- Must NOT depend on any controller header, WAL type, capnp type, or transport — the core stays pure.
|
||||
|
||||
### Revalidation Triggers
|
||||
Dependent specs (`raft-commit-safety`, `raft-sim-conformance`, `ha-term-safety`) must re-check integration if any of the following change:
|
||||
- The signature or semantics of any of the four new callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`).
|
||||
- The signature or return semantics of `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`.
|
||||
- The 1-based-index ↔ WAL-LSN identity, or the "term `0` means absent" convention.
|
||||
- The per-peer `next_index[]` / `match_index[]` initialization or update rules.
|
||||
- The placement of `UpdateTerm` before `logOk` on the AppendEntries path.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
The library is a pure, callback-driven state machine (`raft.c`, ~430 lines). `raft_state_t` holds only Raft invariant state; application policy is delegated to `on_vote_eligible`. Persistence is write-only via `on_persist` — **the library never reads storage**; the caller restores `(term, voted_for)` into `raft_config_t` before `raft_init`. This feature follows the same discipline for the log: the library never reads the WAL directly; it reads entry terms only through `on_get_entry_term`, and writes/truncates only through callbacks. The pattern preserved:
|
||||
|
||||
- **Pure / synchronous / deterministic core** — no I/O, no threads, no allocation in the hot path. The log model uses fixed bookkeeping (a `Len(log)` counter `log_len` plus per-peer vectors); the entry **bytes** live in the adapter (WAL) / simulator (in-memory array), reached via callbacks. The library never owns entry storage — consistent with how it already does not own `(term, voted_for)` storage.
|
||||
- **Persist-before-act ordering** — `on_persist` is called before granting a vote / sending RPCs; `on_persist_entry` is called before reporting an append success, identically.
|
||||
- **Status returns, never self-terminate** — the library returns codes; the controller owns liveness and fencing. No new code path aborts.
|
||||
|
||||
### Architecture Pattern & Boundary Map
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph PureCore["src/lib/raft/raft.c — PURE CORE (this spec)"]
|
||||
APP["raft_log_append()<br/>ClientRequest (tla:246)"]
|
||||
SEND["AppendEntries emit<br/>(tla:204) next/match index"]
|
||||
REQ["raft_on_append_entries_request()<br/>HandleAppendEntriesRequest (tla:327)<br/>UpdateTerm (tla:406) first"]
|
||||
REPLY["raft_on_append_entries_reply()<br/>HandleAppendEntriesResponse (tla:393)"]
|
||||
end
|
||||
subgraph CBs["raft_callbacks_t — NEW seam (verbatim contract)"]
|
||||
C1["on_persist_entry(index,term,entry,len)"]
|
||||
C2["on_get_entry_term(index) -> term|0"]
|
||||
C3["on_send_append_entries(peer,prevIdx,prevTerm,leaderCommit,...)"]
|
||||
C4["on_truncate_from(index)"]
|
||||
end
|
||||
subgraph Consumers["ADAPTERS — implement callbacks (NOT this spec's heavy wiring)"]
|
||||
CTRL["controller adapter (ha-term-safety):<br/>WAL hdr.term=ha->term, repl envelope,<br/>recovery read-back, (term,idx) route"]
|
||||
SIM["raft_sim.c: in-memory log,<br/>synchronous routing (this spec)"]
|
||||
end
|
||||
APP --> C1
|
||||
SEND --> C3
|
||||
REQ --> C2 & C1 & C4
|
||||
REPLY -.updates.-> SEND
|
||||
C1 & C2 & C3 & C4 --> CTRL
|
||||
C1 & C2 & C3 & C4 --> SIM
|
||||
```
|
||||
|
||||
**Architecture Integration**:
|
||||
- **Selected pattern**: Pure state machine + callback ports (Hexagonal/Ports-and-Adapters), already established by the election core. New log actions are new ports; adapters (controller, simulator) implement them.
|
||||
- **Domain boundaries**: protocol decisions (what to persist/send/truncate, accept/reject/conflict) stay in the library; I/O (WAL write, transport, recovery, in-memory backing) stays in adapters.
|
||||
- **Existing patterns preserved**: `raft_config_t` extensibility, persist-before-act, NULL-checked optional callbacks, status-code returns, `XCE_TESTING`-gated simulator.
|
||||
- **Steering compliance**: matches `src/lib/CLAUDE.md` (free-standing C, gnu17), `ha-async-io.md` (no blocking in the core; async lives in adapters), `testing.md` (TDD: simulator/test first), "controller never self-terminates" (library returns statuses only).
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice / Version | Role in Feature | Notes |
|
||||
|-------|------------------|-----------------|-------|
|
||||
| Library core | C (gnu17), `errno.h`/`stdint.h`/`string.h` only | Pure log state machine | Free-standing; no controller/capnp deps |
|
||||
| Callbacks | `raft_callbacks_t` function pointers | Seam for I/O | 4 new required callbacks |
|
||||
| Verification | `raft_sim.c` (XCE_TESTING) + `test_raft.c` | Deterministic Log Matching | In-memory log backing |
|
||||
| Adapter (downstream) | controller `persist.c`/`repl.c` (`ha-term-safety`) | WAL/transport I/O | Not built here |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Modified Files
|
||||
- `src/lib/raft/raft.h` — Add log-model fields to `raft_state_t` (`log_len`, `next_index[RAFT_MAX_PEERS]`, `match_index[RAFT_MAX_PEERS]`); add the four callbacks to `raft_callbacks_t`; declare `raft_log_append`, `raft_on_append_entries_request`, `raft_on_append_entries_reply`; add an `raft_append_result_t` reply struct; extend the simulator section (an in-memory log per `raft_sim_node_t`, an AppendEntries RPC buffer/route).
|
||||
- `src/lib/raft/raft.c` — Implement the three new entry points + a private AppendEntries emit helper; add `on_persist_entry` ordering on `raft_init` validation; apply `UpdateTerm` on the AppendEntries path before `logOk`; initialize `next_index`/`match_index` in `raft_start_election`/`on_become_leader` path.
|
||||
- `src/lib/raft/raft_sim.c` — In-memory per-node log; wire the four callbacks; add `raft_sim_deliver_append_entries` (synchronous request→reply routing honoring partitions); drop/dup/reorder + restart drivers; the Log Matching assertion helper.
|
||||
- `src/lib/testsuite/test_raft.c` — New tests for each entry point and the simulator invariants (TDD-first).
|
||||
|
||||
### Directory Structure (unchanged)
|
||||
```
|
||||
src/lib/raft/
|
||||
├── raft.h # + log fields, 4 callbacks, 3 entry points, sim log
|
||||
├── raft.c # + ClientRequest / AppendEntries / Handle* / UpdateTerm-on-AE
|
||||
└── raft_sim.c # + in-memory log, AE routing, Log Matching assert
|
||||
src/lib/testsuite/
|
||||
└── test_raft.c # + log-replication + Log Matching tests
|
||||
```
|
||||
|
||||
> Each file keeps one responsibility: `raft.c` = pure protocol, `raft_sim.c` = deterministic harness + in-memory adapter, `test_raft.c` = assertions. No new file is introduced; the existing module pattern is extended.
|
||||
|
||||
## System Flows
|
||||
|
||||
### AppendEntries: leader emit → follower handle → leader reply (TLA+ actions in nodes)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant L as Leader (raft.c)
|
||||
participant LC as Leader adapter (cb)
|
||||
participant FC as Follower adapter (cb)
|
||||
participant F as Follower (raft.c)
|
||||
Note over L: raft_log_append(v) — ClientRequest tla:246
|
||||
L->>LC: on_persist_entry(idx, term, v)
|
||||
Note over L: AppendEntries tla:204<br/>prevLogIndex=nextIndex[j]-1
|
||||
L->>LC: on_send_append_entries(j, prevIdx, prevTerm, leaderCommit, entry)
|
||||
LC-->>FC: (transport / sim route)
|
||||
Note over F: raft_on_append_entries_request()<br/>UpdateTerm tla:406 FIRST (G5)
|
||||
F->>FC: on_get_entry_term(prevIdx) %% logOk (G2)
|
||||
alt term < currentTerm OR (follower AND not logOk)
|
||||
F-->>L: reply success=false, matchIndex=0
|
||||
else conflict: local[idx].term != entry.term (G3)
|
||||
F->>FC: on_truncate_from(idx)
|
||||
F->>FC: on_persist_entry(idx, term, entry)
|
||||
F-->>L: reply success=true, matchIndex=idx
|
||||
else no-conflict append OR already-have
|
||||
F->>FC: on_persist_entry(idx, term, entry) %% append case
|
||||
F-->>L: reply success=true, matchIndex=prevIdx+len
|
||||
end
|
||||
Note over L: raft_on_append_entries_reply() tla:393
|
||||
alt success
|
||||
Note over L: nextIndex[j]=matchIndex+1; matchIndex[j]=reply.matchIndex
|
||||
else failure
|
||||
Note over L: nextIndex[j]=Max(nextIndex[j]-1, 1) %% walk back
|
||||
end
|
||||
```
|
||||
|
||||
Flow-level decisions:
|
||||
- **UpdateTerm fires first** on the request path (before `logOk`), so a strictly-higher-term request is never spuriously rejected for a log mismatch (R6.4, closes G5).
|
||||
- **`logOk`** is computed via `on_get_entry_term(prevLogIndex)`; a return of `0` is the "absent" sentinel consistent with `prevLogTerm == 0` when `prevLogIndex == 0` (R1.6, R4.1).
|
||||
- **Conflict truncation** deletes `index..end` via `on_truncate_from` then appends in the same handler invocation (R4.5/R4.6, closes G3) — an intentional, safe collapse of `raft.tla`'s truncate-only conflict step plus the following no-conflict append; see **Note on conflict branch** below.
|
||||
- **Commit is opaque** — `leaderCommit` is passed through untouched; this spec never advances commit (R3.4, R5.4).
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Design Elements | raft.tla action |
|
||||
|-------------|---------|-----------------|-----------------|
|
||||
| 1.1–1.6 | Library log model, per-entry term | `raft_state_t.log_len`; `on_get_entry_term`; `on_persist_entry` | `ClientRequest` (246), `log[i].term` |
|
||||
| 2.1–2.5 | ClientRequest append | `raft_log_append()` | `ClientRequest` (246) |
|
||||
| 3.1–3.5 | AppendEntries emit, next/match | `next_index[]`/`match_index[]`; `on_send_append_entries` | `AppendEntries` (204), `BecomeLeader` (229) |
|
||||
| 4.1–4.7 | logOk / reject / step-down / accept / conflict | `raft_on_append_entries_request()`; `on_get_entry_term`; `on_truncate_from`; `on_persist_entry` | `HandleAppendEntriesRequest` (327) |
|
||||
| 5.1–5.4 | reply advance/decrement | `raft_on_append_entries_reply()` | `HandleAppendEntriesResponse` (393) |
|
||||
| 6.1–6.5 | UpdateTerm on any RPC | term-adoption block in request handler (before logOk); `on_persist`; `on_become_follower` | `UpdateTerm` (406) |
|
||||
| 7.1–7.7 | Callback contract + pure split | `raft_callbacks_t` additions; `raft_init` validation; adapter description | (seam, not a TLA+ action) |
|
||||
| 8.1–8.6 | Simulator Log Matching | `raft_sim.c` AE routing, in-memory log, assert helpers | Log Matching invariant |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|
||||
|-----------|-------|--------|--------------|------------------|-----------|
|
||||
| Log model (`raft_state_t`) | Library core | Per-entry term + per-peer progress | 1, 3 | election core (P0) | State |
|
||||
| `raft_log_append` | Library core | ClientRequest append | 2 | on_persist_entry (P0) | Service |
|
||||
| `raft_on_append_entries_request` | Library core | Handle+logOk+conflict+UpdateTerm | 4, 6 | on_get_entry_term, on_truncate_from, on_persist_entry, on_persist, on_become_follower (P0) | Service |
|
||||
| `raft_on_append_entries_reply` | Library core | Advance/decrement progress | 5 | (internal state) | Service |
|
||||
| New callbacks | Seam | I/O ports | 7 | adapters (P0) | Service |
|
||||
| Simulator | Verification | Deterministic Log Matching | 8 | all of the above (P0) | State |
|
||||
|
||||
### Library Core
|
||||
|
||||
#### Log model additions to `raft_state_t`
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Hold the library's view of log length and per-peer replication progress (TLA+ `logVars` length + `leaderVars`) |
|
||||
| Requirements | 1.1, 1.2, 3.1 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `uint64_t log_len;` — `Len(log)`; the highest 1-based index the library believes is persisted locally. Incremented on append, set by truncate.
|
||||
- `uint64_t next_index[RAFT_MAX_PEERS];` — per-peer `nextIndex` (1-based).
|
||||
- `uint64_t match_index[RAFT_MAX_PEERS];` — per-peer `matchIndex` (0 = nothing matched).
|
||||
- The library stores **no entry bytes and no per-entry term array** — entry terms are obtained via `on_get_entry_term`; bytes live in the adapter/simulator. This keeps `raft_state_t` fixed-size and the core pure (consistent with not storing `(term, voted_for)` bytes).
|
||||
- Invariant: `match_index[j] ≤ next_index[j] - 1` for every peer; `log_len` is the 1:1 WAL LSN.
|
||||
|
||||
**State Management**
|
||||
- `log_len` restored at init from the adapter's recovered WAL length (added to `raft_config_t` as `initial_log_len`, default 0; the library never reads storage).
|
||||
- `next_index[j] = log_len + 1`, `match_index[j] = 0` set when the node becomes leader (mirrors `BecomeLeader`, tla:229).
|
||||
|
||||
#### `raft_log_append` — ClientRequest (raft.tla:246)
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Append one entry stamped with `currentTerm` on the leader |
|
||||
| Requirements | 2.1, 2.2, 2.3, 2.4, 2.5 |
|
||||
|
||||
##### Service Interface
|
||||
```c
|
||||
/*
|
||||
* raft_log_append: leader appends one client entry (raft.tla ClientRequest, 246).
|
||||
* Stamps the entry with s->term, assigns index Len(log)+1, persists it via
|
||||
* on_persist_entry BEFORE returning success (persist-before-act).
|
||||
* Does NOT advance any commit index (owned by raft-commit-safety).
|
||||
*
|
||||
* out_index: receives the assigned 1-based index (== WAL LSN) on success.
|
||||
* Returns RAFT_OK on success; -EPERM if role != RAFT_LEADER; the on_persist_entry
|
||||
* return is propagated on persist failure (no log mutation on failure).
|
||||
*/
|
||||
int raft_log_append(raft_state_t *s, const void *entry, size_t len,
|
||||
uint64_t *out_index);
|
||||
```
|
||||
- Preconditions: `s->role == RAFT_LEADER`.
|
||||
- Postconditions: `log_len` incremented by 1; `*out_index == log_len`; entry persisted at `(log_len, s->term)`.
|
||||
- Invariants: appended entry term == `s->term`; no commit-index change.
|
||||
|
||||
#### `raft_on_append_entries_request` — HandleAppendEntriesRequest (raft.tla:327) + UpdateTerm (406)
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Apply UpdateTerm first, then logOk + reject/step-down/accept (already-have / conflict-truncate / append) |
|
||||
| Requirements | 4.1–4.7, 6.1–6.5 |
|
||||
|
||||
##### Service Interface
|
||||
```c
|
||||
/* Result of handling an AppendEntries request — the adapter sends this back. */
|
||||
typedef struct {
|
||||
uint64_t reply_term; /* current term to return to the leader */
|
||||
int success; /* 1 = accepted, 0 = rejected */
|
||||
uint64_t match_index; /* on success: prevLogIndex + len(entries); else 0 */
|
||||
} raft_append_result_t;
|
||||
|
||||
/*
|
||||
* raft_on_append_entries_request: handle an incoming AppendEntries / heartbeat.
|
||||
* Step order (must not be reordered):
|
||||
* 1. UpdateTerm (raft.tla:406, G5): if term > s->term, adopt term, clear vote,
|
||||
* step down to FOLLOWER, on_persist(term,-1), fire on_become_follower if was
|
||||
* leader. Applied on EVERY RPC including a 0-entry heartbeat, BEFORE logOk.
|
||||
* 2. logOk (raft.tla:328, G2): prevLogIndex==0 || (prevLogIndex<=log_len &&
|
||||
* on_get_entry_term(prevLogIndex)==prevLogTerm).
|
||||
* 3. Reject (raft.tla:333): term<s->term || (term==s->term && FOLLOWER && !logOk)
|
||||
* -> success=0, match_index=0.
|
||||
* 4. Candidate step-down (raft.tla:346): term==s->term && CANDIDATE -> FOLLOWER.
|
||||
* 5. Accept (raft.tla:351), index = prevLogIndex+1:
|
||||
* already-have: entries empty, or local[index].term == entry.term -> success.
|
||||
* conflict (raft.tla:375, G3): local[index] exists, term differs ->
|
||||
* on_truncate_from(index), then on_persist_entry(index, entry).
|
||||
* append (raft.tla:383): log_len == prevLogIndex -> on_persist_entry(index).
|
||||
* entries_len is 0 (heartbeat) or 1 (this spec, like raft.tla).
|
||||
* Returns RAFT_OK; fills *out.
|
||||
*/
|
||||
int raft_on_append_entries_request(raft_state_t *s, int leader_idx,
|
||||
uint64_t term, uint64_t prev_log_index,
|
||||
uint64_t prev_log_term, uint64_t leader_commit,
|
||||
const void *entry, size_t entry_len,
|
||||
uint64_t entry_term, int entries_len,
|
||||
raft_append_result_t *out);
|
||||
```
|
||||
- Preconditions: none beyond a valid `s`.
|
||||
- Postconditions: local log matches the leader through `match_index` on success; divergent suffix truncated on conflict; term adopted if higher.
|
||||
- Invariants: `on_truncate_from` is always followed by re-`on_persist_entry` of the leader's entry; commit is never advanced here (`leader_commit` is opaque, passed for the downstream commit spec to consume — ignored now).
|
||||
|
||||
> **Note on `leader_commit`**: accepted into the signature now (so the wire/sim shape is stable for `raft-commit-safety`) but **not acted on** in this spec — R3.4/R5.4. This avoids a later signature change (a Revalidation Trigger).
|
||||
|
||||
> **Note on conflict branch (intentional, safe divergence from `raft.tla`)**: `raft.tla` (lines 375–388) models the conflict as a **truncate-only** step — `HandleAppendEntriesRequest` deletes the conflicting suffix and returns, and the leader's entry is appended on a *later* AppendEntries via the no-conflict branch (raft.tla:383). XCE deliberately **collapses** the truncate and the immediately-following no-conflict append into a single handler invocation: `on_truncate_from(index)` **then** `on_persist_entry(index, term, entry)`. This is a safe optimization — `raft.tla` itself notes implementations may batch these steps — and Log Matching is preserved because any entry kept after truncation still matches the leader's `(term, index)`, and the freshly-persisted entry is exactly the leader's. On this conflict-and-append path the returned `match_index` is `prevLogIndex + len(entries)` (= `index`), identical to the value the success/no-conflict branch returns, so the leader's `nextIndex`/`matchIndex` accounting is unchanged.
|
||||
|
||||
#### `raft_on_append_entries_reply` — HandleAppendEntriesResponse (raft.tla:393)
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Advance per-peer progress on success; walk back `nextIndex` on failure |
|
||||
| Requirements | 5.1, 5.2, 5.3, 5.4 |
|
||||
|
||||
##### Service Interface
|
||||
```c
|
||||
/*
|
||||
* raft_on_append_entries_reply: process a follower's AppendEntries response
|
||||
* (raft.tla:393). Applied only when reply_term == s->term (stale-term drop, 415).
|
||||
* success: next_index[peer]=match_index+1; match_index[peer]=match_index.
|
||||
* failure: next_index[peer]=Max(next_index[peer]-1, 1); match_index unchanged.
|
||||
* Does NOT advance commit (owned by raft-commit-safety).
|
||||
* Returns RAFT_OK.
|
||||
*/
|
||||
int raft_on_append_entries_reply(raft_state_t *s, int peer_idx, int success,
|
||||
uint64_t match_index, uint64_t reply_term);
|
||||
```
|
||||
- Preconditions: `s->role == RAFT_LEADER` for the update to matter (stale replies are ignored).
|
||||
- Postconditions: `next_index`/`match_index` for `peer_idx` updated per branch.
|
||||
- Invariants: `next_index[peer] ≥ 1` always (floor); commit untouched.
|
||||
|
||||
#### AppendEntries emit helper (raft.tla:204)
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Compute `prevLogIndex`/`prevLogTerm` and emit ≤1 entry via `on_send_append_entries` |
|
||||
| Requirements | 3.1, 3.2, 3.3, 3.4, 3.5 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- For peer `j`: `prevLogIndex = next_index[j] - 1`; `prevLogTerm = (prevLogIndex > 0) ? on_get_entry_term(prevLogIndex) : 0`.
|
||||
- `lastEntry = Min(log_len, next_index[j])`; send `SubSeq(log, next_index[j], lastEntry)` — at most one entry; a heartbeat when `next_index[j] > log_len`.
|
||||
- Calls `on_send_append_entries(ctx, j, prevLogIndex, prevLogTerm, leaderCommit, /* entries handle */ ...)`. `leaderCommit` is supplied opaquely by the caller (the controller/sim), since commit lives outside this spec.
|
||||
- Emitted only while `s->role == RAFT_LEADER` (R3.5).
|
||||
|
||||
> Exposed as `raft_send_append_entries(raft_state_t *s, int peer_idx, uint64_t leader_commit)` so the controller/simulator can drive replication per-peer (fan-out scheduling itself is an adapter concern).
|
||||
|
||||
### Seam — `raft_callbacks_t` additions (verbatim Authoritative shared contract)
|
||||
|
||||
These are appended to `raft_callbacks_t` and validated **required** in `raft_init`. Reproduced **verbatim** from the brief so downstream specs depend on the exact shapes:
|
||||
|
||||
```c
|
||||
/* REQUIRED: durably append/overwrite the log entry at `index` with `term`. */
|
||||
int on_persist_entry(void *ctx, uint64_t index, uint64_t term,
|
||||
const void *entry, size_t len);
|
||||
|
||||
/* REQUIRED: term of the local entry at `index` (0 if absent). */
|
||||
uint64_t on_get_entry_term(void *ctx, uint64_t index);
|
||||
|
||||
/* REQUIRED: transmit one AppendEntries to peer_idx with prevLog{Index,Term},
|
||||
* leaderCommit, and an entries handle. */
|
||||
int on_send_append_entries(void *ctx, int peer_idx,
|
||||
uint64_t prev_log_index, uint64_t prev_log_term,
|
||||
uint64_t leader_commit /*, entries handle */ ...);
|
||||
|
||||
/* REQUIRED: delete local entries at index..end. */
|
||||
int on_truncate_from(void *ctx, uint64_t index);
|
||||
```
|
||||
|
||||
**Contract rules (R7.5)**: the library decides *what / whether* to persist, send, or truncate; the callback only performs I/O. `on_get_entry_term` returning `0` is the canonical "absent" signal (used by `logOk` and by `prevLogTerm` computation). All four are NULL-checked in `raft_init` and required (return `-EINVAL` if NULL), matching how the election-required callbacks are validated today.
|
||||
|
||||
### Controller adapter (seam only — heavy wiring is `ha-term-safety`)
|
||||
|
||||
This spec describes the adapter **only at the seam**; the full controller integration is the downstream `ha-term-safety` consumer.
|
||||
|
||||
- `on_persist_entry` → controller WAL write that stamps `hdr.term` from `ha->term` (closing G1 at the byte level), at the LSN equal to `index`.
|
||||
- `on_get_entry_term` → read the recorded `hdr.term` for a given LSN (from the in-memory WAL index / recovery read-back).
|
||||
- `on_send_append_entries` → enqueue a repl envelope carrying `prevLogIndex`, `prevLogTerm`, the entry, and its term, onto the existing repl transport.
|
||||
- `on_truncate_from` → truncate the WAL at/after the LSN, replacing the LSN-only reconcile in `ha_convergence.c` with the library's `(term, index)` decision.
|
||||
- On recovery, the controller restores `log_len` (= recovered WAL length) and each entry's term, feeding `on_get_entry_term`; `initial_log_len` is passed into `raft_config_t`.
|
||||
|
||||
## Data Models
|
||||
|
||||
### Domain Model
|
||||
- **Log entry (logical)**: `(index: uint64, term: uint64, value: bytes)`. The library holds only `index` accounting (`log_len`) and obtains `term` via callback; `value` bytes are owned by the adapter (WAL) / simulator (in-memory array). Index is 1-based, identical to the controller WAL LSN.
|
||||
- **Per-peer progress**: `(next_index: uint64, match_index: uint64)` per peer, the TLA+ `leaderVars`.
|
||||
|
||||
### Logical Data Model
|
||||
- `raft_state_t` gains `log_len`, `next_index[RAFT_MAX_PEERS]`, `match_index[RAFT_MAX_PEERS]`.
|
||||
- `raft_config_t` gains `initial_log_len` (restored WAL length at boot; 0 on first boot — the library never reads storage, consistent with `initial_term`/`initial_voted_for`).
|
||||
- `raft_append_result_t` is the request-handler reply payload.
|
||||
- **No on-disk format change**: the WAL header already carries a `hdr.term` field (today always 0); this feature only requires the adapter to write the real term. The library imposes no new persisted layout (R7 in requirements / unchanged on-disk format per `ha-term-safety` R1.7).
|
||||
|
||||
### Data Contracts & Integration
|
||||
- **Entries handle**: `on_send_append_entries` and `raft_on_append_entries_request` carry one entry as `(const void *entry, size_t len, uint64_t entry_term)` plus an `entries_len ∈ {0,1}`. This mirrors `raft.tla`'s "up to 1 entry" simplification and keeps the simulator and controller envelope shapes identical.
|
||||
- **Term `0` sentinel**: `on_get_entry_term` returns `0` for an absent index; legacy pre-feature entries with a zero term are handled by the controller adapter (mapped to the lowest-known term) per `ha-term-safety` R1.8 — out of this library's scope, but the `0`-means-absent convention is honored consistently here.
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Strategy
|
||||
The library returns status codes only and **never aborts** (preserves "controller never self-terminates"). Callback failures propagate as negative returns; the caller decides liveness.
|
||||
|
||||
### Error Categories and Responses
|
||||
- **Caller misuse**: `raft_log_append` while not leader → `-EPERM`, no log change. NULL required callback at `raft_init` → `-EINVAL`.
|
||||
- **Persist failure**: `on_persist_entry` non-zero return → propagated; the library does **not** increment `log_len` on a failed append (no phantom entry), preserving the persist-before-act guarantee.
|
||||
- **Truncate failure**: `on_truncate_from` non-zero return on the conflict branch → reported as a non-success request result so the leader retries (the follower has not corrupted its log; it simply did not converge this round).
|
||||
- **Stale RPCs**: a request with `term < currentTerm` is rejected (success=0, matchIndex=0); a reply with `reply_term < currentTerm` is ignored (DropStaleResponse, tla:415).
|
||||
- **Higher-term RPC**: always adopted first (UpdateTerm) — never rejected for a log mismatch (R6.4).
|
||||
|
||||
### Monitoring
|
||||
Reuse the optional `on_trace` callback for new events: `"ae_recv"`, `"ae_reject_stale"`, `"ae_reject_logmismatch"`, `"ae_conflict_truncate"`, `"ae_append"`, `"term_adopt_ae"`, `"ae_reply_advance"`, `"ae_reply_backoff"`. NULL-checked exactly like existing trace calls.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
TDD-mandatory (`testing.md`): tests and simulator drivers are written **before** the `raft.c` implementation (RED → GREEN). All tests are `XCE_TESTING`-gated and live in `src/lib/testsuite/test_raft.c` driving `raft_sim.c`.
|
||||
|
||||
### Unit Tests (entry points, against in-memory sim log)
|
||||
1. `raft_log_append` on a leader stamps `currentTerm`, assigns `Len(log)+1`, calls `on_persist_entry` once, returns the index (R2.1–2.4); on a follower returns `-EPERM` with no log change (R2.2).
|
||||
2. `raft_on_append_entries_request` `logOk`: accepts when `prevLogIndex==0`; accepts when `on_get_entry_term(prevLogIndex)==prevLogTerm`; rejects (success=0, matchIndex=0) when terms differ (R4.1, R4.2 — G2).
|
||||
3. Conflict branch: a follower with `(index=k, term=A)` receiving `(index=k, term=B)` calls `on_truncate_from(k)` then `on_persist_entry(k, B)` and reports success with `matchIndex=k` (R4.5, R4.6 — G3).
|
||||
4. `raft_on_append_entries_reply`: success sets `next_index=matchIndex+1`, `match_index=reply`; failure decrements `next_index` flooring at 1; stale `reply_term` ignored (R5.1–5.3).
|
||||
5. `UpdateTerm` on AppendEntries: a follower/leader at term T receiving an AE/heartbeat at term T+1 adopts T+1, clears vote, persists, and (if was leader) fires `on_become_follower` — applied before `logOk`, including a 0-entry heartbeat (R6.1–6.4 — G5).
|
||||
|
||||
### Integration / Simulator Tests (Log Matching invariant)
|
||||
1. Two-phase replication: leader appends 3 entries, replicates to a quorum; assert every node's prefix is byte-identical (Log Matching, R8.4).
|
||||
2. Partition + heal: isolate a follower during several appends, then heal; assert AppendEntries walks `nextIndex` back, the follower catches up, and Log Matching holds (R8.1, R8.3).
|
||||
3. Divergent-suffix truncation: hand a follower a manufactured divergent tail (same indices, different term), replicate from a higher-term leader; assert the follower truncates and converges to the leader's prefix (R8.5 — G3).
|
||||
4. Chaotic churn / G5 regression: a stale-term node (term lagging) rejoins under a higher-term leader; assert it adopts the higher term from the AppendEntries/heartbeat path and does **not** re-elect at a reused term (R8.6 — closes the bug-851 G5 mechanism in the deterministic harness).
|
||||
|
||||
### Message-fault Tests
|
||||
1. Drop, duplicate, and reorder AppendEntries requests/replies in the sim queue; assert idempotent convergence and Log Matching after the faults clear (R8.3) — the already-have accept branch (R4.4) makes a duplicated AE a no-op.
|
||||
143
.kiro/specs/raft-log-replication/requirements.md
Normal file
143
.kiro/specs/raft-log-replication/requirements.md
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
`src/lib/raft/raft.{c,h}` is today an **election-only** Raft library: it implements terms, votes, leader election, and PreVote with a pure callback interface (`raft_callbacks_t`, `raft_state_t`) and a deterministic simulator (`raft_sim.c`). It has **no log, no AppendEntries, and no commit**. The replicated log is instead hand-rolled across the controller (`persist.c` WAL, `repl.c` catch-up streaming, `ha.c`, `ha_convergence.c`) and **violates the log invariants of `raft/raft.tla`** (Ongaro's canonical Raft TLA+ spec). That split root-caused **bug-851**: silent committed-prefix divergence under chaotic failover. The authoritative gap analysis is `.kiro/specs/ha-term-safety/TLA-gap-analysis.md` (gaps G1–G6).
|
||||
|
||||
This feature extends the library so it **owns the replicated log**, strictly mirroring the `raft.tla` log actions, while keeping the core **pure / synchronous / deterministic** (no I/O, no threads) so it remains independently verifiable in the simulator against the Raft safety invariants. All side effects flow through new callbacks that a controller adapter implements over the existing WAL/repl transport. The feature closes exactly the four **log-half** gaps:
|
||||
|
||||
- **G1** — per-entry term is dead today (WAL `hdr.term` always 0; `p->term` never set from `ha->term`). The library log model gives every entry a real `uint64_t` term, set to the leader's current term at append time (`raft.tla` `ClientRequest`, line 246).
|
||||
- **G2** — replication matches LSN contiguity only; there is no `prevLogTerm` check. The library enforces the `logOk` predicate (`raft.tla` `AppendEntries` line 204 / `HandleAppendEntriesRequest` line 327): `prevLogIndex == 0 ∨ log[prevLogIndex].term == prevLogTerm`.
|
||||
- **G3** — no `(term, index)` conflict truncation; reconcile truncates by a committed-boundary heuristic and cannot detect same-index/different-term divergence. The library implements the TLA+ conflict branch: delete the divergent entry and everything after it.
|
||||
- **G5** — a node adopts a higher term only on **vote** RPCs; the AppendEntries/heartbeat path never advances `currentTerm`. The library applies `raft.tla` `UpdateTerm` (line 406) on **every** incoming RPC including AppendEntries/heartbeat: a strictly higher term ⇒ adopt term, clear vote, step down to follower.
|
||||
|
||||
Commit-index advance (`AdvanceCommitIndex`, G4) and `(lastLogTerm, lastLogIndex)` vote freshness (G6) are **out of scope** and are closed by the downstream `raft-commit-safety` spec. The full conformance harness is `raft-sim-conformance`. The controller-side wiring of WAL / commit-barrier / snapshot onto these callbacks is owned by `ha-term-safety`. This spec's contract (the new callbacks and entry points below) is consumed **verbatim** by all three downstream specs.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**:
|
||||
- A per-node log model **inside the library** where every entry carries a `uint64_t` term (G1), with the log index 1-based (TLA+) and mapping 1:1 to the controller WAL LSN.
|
||||
- `ClientRequest` append: `raft_log_append(...)` appends `[term ↦ currentTerm, value]` on the leader (`raft.tla` line 246).
|
||||
- `AppendEntries` send: per-peer `nextIndex` / `matchIndex` leader state; emission of `prevLogIndex`, `prevLogTerm`, ≤1 entry, and `leaderCommit` via the `on_send_append_entries` callback (`raft.tla` line 204).
|
||||
- `HandleAppendEntriesRequest`: the `logOk` `prevLogTerm` match (G2), reject (stale term / `¬logOk`), candidate→follower on equal term, and the accept branch's three sub-cases — already-have, conflict-truncate, append-by-`(term, index)` (G3) (`raft.tla` line 327).
|
||||
- `HandleAppendEntriesResponse`: success advances `nextIndex` / `matchIndex`; failure decrements `nextIndex` (floored at 1) (`raft.tla` line 393).
|
||||
- `UpdateTerm` applied on **every** incoming RPC including AppendEntries/heartbeat (G5) (`raft.tla` line 406).
|
||||
- The shared callback contract: `on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from` (pure protocol decisions in the library; I/O in the controller adapter).
|
||||
- The controller adapter only at the seam: it stamps the WAL `hdr.term` from `ha->term`, carries the term in the repl envelope, reads it back on recovery, and routes `(term, index)` match/truncate decisions back into the library.
|
||||
- Simulator coverage in `raft_sim.c` that drives AppendEntries across partitions, message drop/dup/reorder, and restarts, asserting the **Log Matching** invariant and divergent-suffix truncation.
|
||||
|
||||
- **Out of scope** (owned elsewhere):
|
||||
- `AdvanceCommitIndex` / commit-index advance and the Figure-8 current-term commit rule (G4) → `raft-commit-safety`.
|
||||
- `RequestVote` / PreVote freshness becoming `(lastLogTerm, lastLogIndex)` instead of LSN-only (G6) → `raft-commit-safety`.
|
||||
- The full conformance / TLC trace-check harness → `raft-sim-conformance`.
|
||||
- Configuration-change markers in the unified log + the upgrade snapshot → `ha-term-safety`.
|
||||
- Async vtask I/O, fan-out replication scheduling, the synchronous commit barrier, persist-before-ACK durability, and snapshot transport → controller callback adapters (`ha-term-safety`), not the pure core.
|
||||
- The heavy controller wiring of WAL / repl / recovery onto the callbacks → `ha-term-safety` (this spec defines only the seam contract and a thin adapter description).
|
||||
|
||||
- **Adjacent expectations**:
|
||||
- Builds on the implemented election core (`raft-library`); all existing election callbacks and behaviour are unchanged.
|
||||
- Preserves as policy (must NOT be canonicalized away): PreVote, priority leadership, persist-before-ACK durability + the synchronous commit barrier (consumed downstream), and "controller never self-terminates" (the library returns statuses; the controller owns liveness/fencing — the library never aborts a process).
|
||||
- Feeds `raft-commit-safety` (commit + freshness), `raft-sim-conformance` (verification harness), and `ha-term-safety` (XCE integration + regression). Its protocol invariants R1–R5 there are realized by this library spec rather than by patching the controller directly.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Library-owned log model with per-entry term (closes G1)
|
||||
|
||||
**Objective:** As a consumer of the raft library, I want the library to own a log where every entry carries the term under which it was created, so that entries from different terms are distinguishable and Raft log-matching becomes possible.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Raft Library shall maintain, per node, a 1-based log index whose value at index `i` includes the `uint64_t` term recorded for the entry at that index, mirroring `raft.tla` `log[i][...].term`.
|
||||
2. The Raft Library shall expose the log index as identical to the controller WAL LSN (index 1 ↔ LSN 1), so callbacks and the adapter address entries by a single shared `(term, index)` identity.
|
||||
3. When the leader appends a client value, the Raft Library shall record the leader's current `currentTerm` as that entry's term, mirroring `raft.tla` `ClientRequest` (`entry == [term ↦ currentTerm[i], value ↦ v]`).
|
||||
4. The Raft Library shall not record a fixed or zero placeholder term for any entry created while a valid term is in effect (the G1 defect).
|
||||
5. While persisting or overwriting an entry, the Raft Library shall delegate the durable write to the `on_persist_entry(ctx, index, term, entry, len)` callback and shall not itself perform any I/O.
|
||||
6. When the library needs the term of a local entry at a given index, it shall obtain it via the `on_get_entry_term(ctx, index)` callback, treating a return of `0` as "no entry at that index" consistent with `raft.tla` `prevLogTerm == 0` when `prevLogIndex == 0`.
|
||||
|
||||
### Requirement 2: ClientRequest append on the leader (raft.tla ClientRequest, line 246)
|
||||
|
||||
**Objective:** As a controller acting as primary, I want a single library entry point to append a new log entry stamped with the current term, so that the library — not hand-rolled controller code — owns log growth.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When `raft_log_append(...)` is called while the node's role is `RAFT_LEADER`, the Raft Library shall append one entry at index `Len(log) + 1` with term equal to `currentTerm`, mirroring `raft.tla` `ClientRequest`.
|
||||
2. If `raft_log_append(...)` is called while the node's role is not `RAFT_LEADER`, then the Raft Library shall reject the append and report a non-success status without mutating the log.
|
||||
3. When appending, the Raft Library shall invoke `on_persist_entry` for the new `(index, term, entry)` before reporting success, preserving the persist-before-act ordering used by the existing election path's `on_persist`.
|
||||
4. The Raft Library shall return the assigned 1-based index to the caller on a successful append, so the controller adapter can map it to the WAL LSN.
|
||||
5. The Raft Library shall not, in this spec, advance any commit index as a side effect of append (commit advance is owned by `raft-commit-safety`).
|
||||
|
||||
### Requirement 3: AppendEntries emission with per-peer next/match index (raft.tla AppendEntries, line 204)
|
||||
|
||||
**Objective:** As a primary replicating to standbys, I want the library to own per-peer replication progress and emit correctly-formed AppendEntries, so that replication carries the `(prevLogIndex, prevLogTerm)` needed for log-matching.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Raft Library shall maintain, per peer, a `nextIndex` and a `matchIndex` (the `raft.tla` leaderVars), initialized on becoming leader to `nextIndex = Len(log) + 1` and `matchIndex = 0` for every peer, mirroring `raft.tla` `BecomeLeader`.
|
||||
2. When the library emits an AppendEntries to peer `j`, it shall compute `prevLogIndex = nextIndex[j] - 1` and `prevLogTerm = (prevLogIndex > 0) ? term-at-prevLogIndex : 0`, mirroring `raft.tla` `AppendEntries`.
|
||||
3. When the library emits an AppendEntries to peer `j`, it shall send at most one entry, constrained by the end of the log (`SubSeq(log, nextIndex[j], Min(Len(log), nextIndex[j]))`), via the `on_send_append_entries(ctx, peer_idx, prevLogIndex, prevLogTerm, leaderCommit, ...)` callback.
|
||||
4. The Raft Library shall pass the leader commit value through `on_send_append_entries` as an opaque `leaderCommit` argument and shall not itself interpret or advance commit (commit handling is owned downstream).
|
||||
5. While the node's role is not `RAFT_LEADER`, the Raft Library shall not emit AppendEntries for that node.
|
||||
|
||||
### Requirement 4: HandleAppendEntriesRequest — logOk, reject, step-down, accept (raft.tla line 327; closes G2, G3)
|
||||
|
||||
**Objective:** As a standby receiving replication, I want the library to accept, reject, or conflict-truncate strictly by `(term, index)` matching, so that Log Matching holds and divergent suffixes are truncated.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When `raft_on_append_entries_request(...)` is processed, the Raft Library shall compute `logOk = (prevLogIndex == 0) ∨ (prevLogIndex ≤ Len(log) ∧ on_get_entry_term(prevLogIndex) == prevLogTerm)`, mirroring the `raft.tla` `logOk` predicate (closes G2).
|
||||
2. If the request term is less than `currentTerm`, or the term equals `currentTerm` and the node is a follower and `¬logOk`, then the Raft Library shall reject the request and report failure with `matchIndex = 0`, mirroring the `raft.tla` reject branch.
|
||||
3. When the request term equals `currentTerm` and the node is a candidate, the Raft Library shall transition the node to follower (return-to-follower branch) without otherwise mutating the log.
|
||||
4. When the request term equals `currentTerm`, the node is a follower, and `logOk` holds, and the incoming entry already matches the local entry at `index = prevLogIndex + 1` (same term, or no entries sent), the Raft Library shall treat the request as already-satisfied and report success with `matchIndex = prevLogIndex + len(entries)` without rewriting the log.
|
||||
5. When `logOk` holds and the local entry at `index = prevLogIndex + 1` exists with a **different** term than the incoming entry, the Raft Library shall delete the divergent entry and all entries after it via `on_truncate_from(ctx, index)`, mirroring the `raft.tla` conflict branch (closes G3).
|
||||
6. When `logOk` holds and the local log ends exactly at `prevLogIndex` (no entry at `prevLogIndex + 1`), the Raft Library shall append the incoming entry via `on_persist_entry` and report success, mirroring the `raft.tla` no-conflict append branch.
|
||||
7. The Raft Library shall return a reply carrying success/failure and the resulting `matchIndex` so the leader can update its per-peer progress, and shall not perform transport I/O itself (the adapter sends the reply).
|
||||
|
||||
### Requirement 5: HandleAppendEntriesResponse — advance/decrement progress (raft.tla line 393)
|
||||
|
||||
**Objective:** As a primary processing replication replies, I want the library to advance or back off per-peer progress, so that a lagging or divergent follower is walked back to the matching point.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When `raft_on_append_entries_reply(...)` reports success, the Raft Library shall set `nextIndex[peer] = matchIndex + 1` and `matchIndex[peer] = reply.matchIndex`, mirroring `raft.tla` `HandleAppendEntriesResponse` success.
|
||||
2. If `raft_on_append_entries_reply(...)` reports failure, then the Raft Library shall set `nextIndex[peer] = Max(nextIndex[peer] - 1, 1)` and leave `matchIndex[peer]` unchanged, mirroring the `raft.tla` failure branch.
|
||||
3. The Raft Library shall apply a reply only when the reply term equals `currentTerm`, ignoring stale-term responses, consistent with `raft.tla` `HandleAppendEntriesResponse` / `DropStaleResponse`.
|
||||
4. The Raft Library shall not, in this spec, recompute or advance any commit index from updated `matchIndex` values (owned by `raft-commit-safety`).
|
||||
|
||||
### Requirement 6: UpdateTerm on every RPC including AppendEntries/heartbeat (raft.tla line 406; closes G5)
|
||||
|
||||
**Objective:** As any cluster member, I want a strictly-higher term on **any** incoming RPC — vote, AppendEntries, or heartbeat — to advance my term and step me down, so that a rejoined node can never act or re-elect at a stale term and two leaders can never share a term.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When an incoming AppendEntries or heartbeat RPC carries a term strictly greater than `currentTerm`, the Raft Library shall set `currentTerm` to that term, clear `voted_for`, and transition to follower, mirroring `raft.tla` `UpdateTerm` (closes G5).
|
||||
2. While adopting a higher term from any RPC, the Raft Library shall persist the new `(term, voted_for)` via the existing `on_persist` callback before processing the rest of the RPC, preserving the durability ordering already used on the vote path.
|
||||
3. When a node that was leader adopts a higher term from an AppendEntries/heartbeat RPC, the Raft Library shall fire `on_become_follower` so the controller HA FSM steps down (mirroring the existing higher-term step-down already wired for vote RPCs).
|
||||
4. The Raft Library shall apply the term-adoption check on the AppendEntries path **before** the `logOk` / accept-reject logic, so a higher-term request is never rejected merely because the local log does not match.
|
||||
5. The Raft Library shall not, on equal or lower term, alter `currentTerm` or `voted_for` from the AppendEntries path (term adoption is strictly-greater-than only), consistent with `raft.tla`.
|
||||
|
||||
### Requirement 7: Shared callback contract and pure-core / adapter split
|
||||
|
||||
**Objective:** As a downstream spec author (`raft-commit-safety`, `raft-sim-conformance`, `ha-term-safety`), I want the new callbacks and entry points fixed verbatim and the library kept pure, so that I can depend on a stable seam without re-deriving the protocol.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Raft Library shall define the required callback `int on_persist_entry(void *ctx, uint64_t index, uint64_t term, const void *entry, size_t len)` to durably append or overwrite the entry at `index`.
|
||||
2. The Raft Library shall define the required callback `uint64_t on_get_entry_term(void *ctx, uint64_t index)` returning the term of the local entry at `index`, or `0` when absent.
|
||||
3. The Raft Library shall define the required callback `int on_send_append_entries(void *ctx, int peer_idx, uint64_t prev_log_index, uint64_t prev_log_term, uint64_t leader_commit, ...)` carrying an entries handle, to transmit one AppendEntries.
|
||||
4. The Raft Library shall define the required callback `int on_truncate_from(void *ctx, uint64_t index)` to delete local entries in `index..end`.
|
||||
5. The Raft Library shall keep all four callbacks free of protocol decisions: the library decides what/whether to persist, send, or truncate; the controller adapter only performs the I/O (WAL write, transport, recovery read-back).
|
||||
6. The Raft Library shall leave all existing election callbacks (`on_send_vote_request`, `on_become_leader`, `on_become_follower`, `on_persist`, `on_schedule_timeout`, `on_cancel_timeout`, `on_start_election_after_delay`, optional `on_vote_eligible` / `on_send_pre_vote_request` / `on_trace`) and their behaviour unchanged.
|
||||
7. The controller adapter shall implement the four callbacks over the existing WAL/repl transport — stamping `hdr.term` from `ha->term`, carrying the term in the repl envelope, reading it back on recovery, and routing `(term, index)` match/truncate decisions to the library — while the heavy controller wiring lives in `ha-term-safety` as a downstream consumer.
|
||||
|
||||
### Requirement 8: Deterministic Log Matching verification in the simulator
|
||||
|
||||
**Objective:** As a maintainer, I want the deterministic simulator to drive replication under adversarial conditions and assert Raft Log Matching, so that the log-half implementation is verifiable without a live cluster.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Raft Simulator shall route AppendEntries requests and responses synchronously between nodes (mirroring its existing vote-RPC routing), honoring per-node partition flags.
|
||||
2. The Raft Simulator shall provide an in-memory per-node log backing the four new callbacks, so a node can append, read entry terms, send, and truncate deterministically.
|
||||
3. The Raft Simulator shall exercise message drop, duplication, and reordering of AppendEntries, plus node restart, while replication proceeds.
|
||||
4. The Raft Simulator shall assert the **Log Matching** invariant after replication: for any two nodes and any index `i`, if both hold an entry at `i` with the same term, then their entire prefixes through `i` are byte-identical.
|
||||
5. When a follower holds a divergent suffix (same index, different term) and then receives the leader's AppendEntries, the Raft Simulator shall assert that the follower truncates the divergent suffix and converges to the leader's prefix.
|
||||
6. While driving chaotic leadership churn (a stale-term node rejoining under a higher-term leader), the Raft Simulator shall assert that the rejoining node adopts the higher term from the AppendEntries/heartbeat path (G5) and does not re-elect at a reused term.
|
||||
34
.kiro/specs/raft-log-replication/research.md
Normal file
34
.kiro/specs/raft-log-replication/research.md
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# Research Log: raft-log-replication
|
||||
|
||||
## Discovery Scope
|
||||
|
||||
Extension of an existing, implemented pure-C library (`src/lib/raft/raft.{c,h}` + `raft_sim.c`). Discovery type: **Extension / integration-focused** — the architecture pattern (pure state machine + callback ports + deterministic simulator) already exists and is authoritative; this feature adds the log half of `raft/raft.tla` to it. No external dependency research needed (the library is free-standing C with only `errno.h`/`stdint.h`/`string.h`).
|
||||
|
||||
## Authoritative Sources
|
||||
|
||||
| Source | Use |
|
||||
|--------|-----|
|
||||
| `.kiro/specs/raft-log-replication/brief.md` | Problem, approach, **Authoritative shared callback contract** (binding, reproduced verbatim in design), TLA+ action→entry-point map |
|
||||
| `raft/raft.tla` | Canonical log actions: `ClientRequest` (246), `AppendEntries` (204), `HandleAppendEntriesRequest` (327), `HandleAppendEntriesResponse` (393), `UpdateTerm` (406), `BecomeLeader` (229) |
|
||||
| `.kiro/specs/ha-term-safety/TLA-gap-analysis.md` | Gaps G1–G6; this spec closes G1/G2/G3/G5 |
|
||||
| `src/lib/raft/raft.{c,h}` | Existing election core style: `raft_config_t` extensibility, persist-before-act, NULL-checked required callbacks, status-code returns, `XCE_TESTING` sim |
|
||||
| `.kiro/steering/roadmap.md` (Raft TLA+ Conformance series) | Family policy: preserve PreVote, priority leadership, sync commit-barrier + persist-before-ACK, controller-never-self-terminates |
|
||||
|
||||
## Key Findings
|
||||
|
||||
1. **The library must NOT store entry bytes.** The election core already declines to store `(term, voted_for)` bytes — the caller restores them via `raft_config_t` and the library writes via `on_persist`. The log follows the same discipline: the library tracks only `log_len` + per-peer `next_index`/`match_index`; entry terms come from `on_get_entry_term`, bytes live in the adapter/simulator. This keeps `raft_state_t` fixed-size and the core pure/deterministic (simulator-verifiable). **Decision**: no per-entry array in `raft_state_t`.
|
||||
|
||||
2. **Index ↔ LSN is 1:1 and 1-based.** `raft.tla` logs are 1-based; the controller WAL LSN is the natural mapping. Settling this in the contract (R1.2) means the controller adapter and simulator address entries identically. `on_get_entry_term(0)` / `prevLogIndex==0` → term `0` is the absent sentinel, matching `raft.tla`'s `prevLogTerm == 0` when `prevLogIndex == 0`.
|
||||
|
||||
3. **UpdateTerm must precede logOk on the AE path (G5).** The bug-851 mechanism: a rejoined standby never adopted the leader's term from heartbeats, re-elected at a stale+1 term, and reused a term → two leaders/term. `raft.tla` `Receive` applies `UpdateTerm` before any handler. **Decision**: the request handler applies term adoption first, unconditionally, including for 0-entry heartbeats — otherwise a higher-term request could be rejected for a log mismatch and never advance the term (R6.4).
|
||||
|
||||
4. **`leader_commit` is carried but not acted on.** Commit advance (G4) is `raft-commit-safety`. To avoid a later signature change (a Revalidation Trigger for all downstream consumers), the AppendEntries entry points accept `leader_commit` opaquely **now** and ignore it. This stabilizes the wire/sim shape across the two specs. **Decision (build-vs-defer)**: include the parameter, defer the behavior.
|
||||
|
||||
5. **`(term, index)` truncation replaces the LSN-only reconcile.** The controller's `ha_convergence.c` truncates by a committed-boundary heuristic (G3) and cannot see same-LSN/different-term divergence. The library's conflict branch (`on_truncate_from(index)` + re-append) is the exact `raft.tla` semantics. The heavy controller swap is `ha-term-safety`; this spec owns the library decision + the simulator proof.
|
||||
|
||||
## Synthesis Outcomes
|
||||
|
||||
- **Generalization found**: the existing "pure core, I/O via callbacks, restore-state-via-config, never-read-storage" pattern generalizes cleanly from election state to log state. No new architectural concept is introduced — only new ports and new pure transitions.
|
||||
- **Build-vs-adopt**: adopt the canonical `raft.tla` algorithm verbatim (it is the verified reference). Build only the C translation + the in-memory simulator backing. No third-party Raft library is imported (the project's constraint is a free-standing, simulator-verifiable core).
|
||||
- **Simplification**: keep the "≤1 entry per AppendEntries" `raft.tla` simplification rather than batching — it minimizes atomic regions, matches the spec exactly, and keeps the simulator's Log Matching assertion tractable. Batching is an adapter-side optimization that can be layered later without changing the core contract.
|
||||
- **Risk**: a future signature change to the four callbacks or the three entry points cascades to three downstream specs (logged as Revalidation Triggers). Mitigated by reproducing the brief's contract verbatim and by accepting `leader_commit` now.
|
||||
23
.kiro/specs/raft-log-replication/spec.json
Normal file
23
.kiro/specs/raft-log-replication/spec.json
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"feature_name": "raft-log-replication",
|
||||
"created_at": "2026-06-21T11:00:00Z",
|
||||
"updated_at": "2026-06-21T11:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true,
|
||||
"note": "Auto-approved (-y mode). Extends election-only src/lib/raft into a strict raft.tla log-half implementation: ClientRequest/AppendEntries/HandleAppendEntriesRequest/HandleAppendEntriesResponse/UpdateTerm-on-any-RPC. Closes bug-851 G1/G2/G3/G5. Authoritative shared callback contract (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from) consumed verbatim by raft-commit-safety, raft-sim-conformance, and ha-term-safety. Boundary: owns src/lib/raft log additions + (term,index) reconcile seam only; NOT commit-index advance, vote freshness, config, or controller WAL/transport internals."
|
||||
}
|
||||
196
.kiro/specs/raft-log-replication/tasks.md
Normal file
196
.kiro/specs/raft-log-replication/tasks.md
Normal file
|
|
@ -0,0 +1,196 @@
|
|||
# Implementation Plan
|
||||
|
||||
> **TDD-mandatory** (`testing.md`): every implementation sub-task is preceded by a RED test in `src/lib/testsuite/test_raft.c` (and the simulator backing in `raft_sim.c`) that fails before the `raft.c` change and passes after. Build/run library tests via `/xce-build lib`.
|
||||
>
|
||||
> **Spec boundary** (applies to ALL tasks): this spec owns the `src/lib/raft` log additions (per-entry term in the library model, `ClientRequest`/`AppendEntries`/`Handle*`/`UpdateTerm`-on-AE) and the `(term, index)` reconcile seam (the four callbacks). It does **NOT** own commit-index advance, `(lastLogTerm,lastLogIndex)` vote freshness, config markers/upgrade snapshot, or the controller's WAL/repl/recovery internals — those are `raft-commit-safety` and `ha-term-safety`. No task here edits any controller file; the controller adapter is described, not built.
|
||||
|
||||
## 1. Header contract: log model, callbacks, entry points
|
||||
|
||||
- [ ] 1.1 Extend `raft.h` with the log model state and config restore field
|
||||
- Add `uint64_t log_len;`, `uint64_t next_index[RAFT_MAX_PEERS];`, `uint64_t match_index[RAFT_MAX_PEERS];` to `raft_state_t`
|
||||
- Add `uint64_t initial_log_len;` to `raft_config_t` (restored WAL length at boot, default 0; library never reads storage)
|
||||
- Observable completion: `raft.h` compiles; `sizeof(raft_state_t)` grows by the new fixed fields; no entry-byte storage added
|
||||
- _Requirements: 1.1, 1.2, 3.1_
|
||||
- _Boundary: raft.h_
|
||||
|
||||
- [ ] 1.2 Add the four new REQUIRED callbacks to `raft_callbacks_t` (verbatim contract)
|
||||
- Declare `on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from` with the exact signatures from design.md / brief
|
||||
- Document each as REQUIRED and the `on_get_entry_term` → `0`-means-absent convention
|
||||
- Observable completion: the four function-pointer members exist in `raft_callbacks_t` with the verbatim signatures; header doc comments match the Authoritative shared contract
|
||||
- _Requirements: 7.1, 7.2, 7.3, 7.4_
|
||||
- _Boundary: raft.h_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 1.3 Declare the three new entry points and `raft_append_result_t`
|
||||
- Declare `raft_append_result_t`; declare `raft_log_append`, `raft_send_append_entries`, `raft_on_append_entries_request`, `raft_on_append_entries_reply` with the design.md signatures
|
||||
- Observable completion: header declares all new entry points; a stub TU including `raft.h` compiles
|
||||
- _Requirements: 2.1, 3.3, 4.7, 5.1_
|
||||
- _Boundary: raft.h_
|
||||
- _Depends: 1.2_
|
||||
|
||||
## 2. Init validation and leader-progress initialization
|
||||
|
||||
- [ ] 2.1 RED+GREEN: `raft_init` requires the four new callbacks
|
||||
- Test: `raft_init` returns `-EINVAL` when any of the four new callbacks is NULL; returns 0 when all present
|
||||
- Implement the NULL checks in `raft_init` (mirroring the existing required-callback checks); restore `s->log_len = cfg->initial_log_len`
|
||||
- Observable completion: new `-EINVAL` test cases pass; existing election init tests still green
|
||||
- _Requirements: 7.1, 7.2, 7.3, 7.4, 1.1_
|
||||
- _Boundary: raft.c, test_raft.c_
|
||||
- _Depends: 1.3_
|
||||
|
||||
- [ ] 2.2 RED+GREEN: leader progress initialized on becoming leader
|
||||
- Test: after a node becomes leader, `next_index[j] == log_len + 1` and `match_index[j] == 0` for every peer (raft.tla BecomeLeader)
|
||||
- Implement the `next_index`/`match_index` initialization on the win path (`raft_start_election` single-node + the multi-node become-leader transition)
|
||||
- Observable completion: assertion on `next_index`/`match_index` after election passes; no regression in election tests
|
||||
- _Requirements: 3.1_
|
||||
- _Boundary: raft.c, test_raft.c_
|
||||
- _Depends: 2.1_
|
||||
|
||||
## 3. ClientRequest — raft_log_append (raft.tla:246)
|
||||
|
||||
- [ ] 3.1 RED: append tests (leader stamps term, follower rejected)
|
||||
- Test: leader `raft_log_append` assigns index `log_len+1`, stamps `currentTerm`, calls `on_persist_entry` exactly once with `(index, term, entry)`, returns the index; follower call returns `-EPERM` with no log change and no callback
|
||||
- Observable completion: tests compile and FAIL (no implementation yet)
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 3.2 GREEN: implement `raft_log_append`
|
||||
- Role guard (`-EPERM` if not leader); persist-before-act (`on_persist_entry` before incrementing `log_len`); set `*out_index`; do NOT advance any commit index
|
||||
- Observable completion: 3.1 tests pass; `log_len` only advances on a successful persist (no phantom entry on persist failure)
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5_
|
||||
- _Boundary: raft.c_
|
||||
- _Depends: 3.1_
|
||||
|
||||
## 4. AppendEntries emission — raft_send_append_entries (raft.tla:204)
|
||||
|
||||
- [ ] 4.1 RED: emit tests (prevLogIndex/prevLogTerm, ≤1 entry, leader-only)
|
||||
- Test: for peer `j`, emit computes `prevLogIndex = next_index[j]-1` and `prevLogTerm` via `on_get_entry_term` (0 when `prevLogIndex==0`), sends ≤1 entry, passes `leader_commit` through opaquely; a non-leader emits nothing
|
||||
- Observable completion: tests compile and FAIL
|
||||
- _Requirements: 3.2, 3.3, 3.4, 3.5_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 4.2 GREEN: implement `raft_send_append_entries`
|
||||
- Compute prev index/term; `lastEntry = Min(log_len, next_index[j])`; one entry or heartbeat; call `on_send_append_entries` with the opaque `leader_commit`; leader-only guard
|
||||
- Observable completion: 4.1 tests pass; heartbeat (0 entries) emitted when `next_index[j] > log_len`
|
||||
- _Requirements: 3.2, 3.3, 3.4, 3.5_
|
||||
- _Boundary: raft.c_
|
||||
- _Depends: 4.1_
|
||||
|
||||
## 5. UpdateTerm-on-AppendEntries (raft.tla:406, closes G5)
|
||||
|
||||
- [ ] 5.1 RED: higher-term AE/heartbeat adopts term before logOk
|
||||
- Test: a follower and a leader at term T receiving an AE (and separately a 0-entry heartbeat) at term T+1 adopt T+1, clear `voted_for`, call `on_persist(T+1,-1)`, and (was-leader case) fire `on_become_follower`; assert this happens even when the local log would fail `logOk`
|
||||
- Observable completion: tests compile and FAIL
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 1.3_
|
||||
|
||||
- [ ] 5.2 GREEN: term-adoption block at the top of the request handler
|
||||
- Implement the `UpdateTerm` block as the FIRST step of `raft_on_append_entries_request`, before `logOk`; persist-before-process; fire `on_become_follower` if was leader; no change on equal/lower term
|
||||
- Observable completion: 5.1 tests pass; the block is unmistakably before `logOk` in `raft.c`
|
||||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5_
|
||||
- _Boundary: raft.c_
|
||||
- _Depends: 5.1_
|
||||
|
||||
## 6. HandleAppendEntriesRequest — logOk / reject / step-down / accept (raft.tla:327, closes G2/G3)
|
||||
|
||||
- [ ] 6.1 RED: logOk + reject + candidate step-down tests
|
||||
- Test: `logOk` true when `prevLogIndex==0` or `on_get_entry_term(prevLogIndex)==prevLogTerm`; reject (success=0, matchIndex=0) on stale term and on `follower && !logOk`; candidate at equal term transitions to follower
|
||||
- Observable completion: tests compile and FAIL
|
||||
- _Requirements: 4.1, 4.2, 4.3_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 5.2_
|
||||
|
||||
- [ ] 6.2 RED: accept branch tests (already-have / conflict-truncate / append)
|
||||
- Test: already-have (empty entries or matching term) → success, no log rewrite, `matchIndex = prevLogIndex + entries_len`; conflict (`local[index].term != entry.term`) → `on_truncate_from(index)` then `on_persist_entry(index, entry)`, success `matchIndex=index`; append (`log_len == prevLogIndex`) → `on_persist_entry(index)`, success
|
||||
- Observable completion: tests compile and FAIL
|
||||
- _Requirements: 4.4, 4.5, 4.6, 4.7_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 6.1_
|
||||
|
||||
- [ ] 6.3 GREEN: implement `raft_on_append_entries_request` (logOk + all branches)
|
||||
- Implement (after the 5.2 UpdateTerm block): `logOk` via `on_get_entry_term`; reject/candidate-step-down/accept; the three accept sub-cases exactly per raft.tla:351-388; fill `raft_append_result_t`; never advance commit (`leader_commit` ignored); never perform transport I/O
|
||||
- Observable completion: 6.1 and 6.2 tests pass; `on_truncate_from` is always followed by re-persist of the leader entry on the conflict branch
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7_
|
||||
- _Boundary: raft.c_
|
||||
- _Depends: 6.2_
|
||||
|
||||
## 7. HandleAppendEntriesResponse — raft_on_append_entries_reply (raft.tla:393)
|
||||
|
||||
- [ ] 7.1 RED: reply advance/decrement/stale-drop tests
|
||||
- Test: success sets `next_index[peer]=matchIndex+1`, `match_index[peer]=reply.matchIndex`; failure sets `next_index[peer]=Max(next_index[peer]-1,1)` (floor at 1), `match_index` unchanged; a reply with `reply_term != currentTerm` is ignored
|
||||
- Observable completion: tests compile and FAIL
|
||||
- _Requirements: 5.1, 5.2, 5.3_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 2.2_
|
||||
|
||||
- [ ] 7.2 GREEN: implement `raft_on_append_entries_reply`
|
||||
- Stale-term drop; success/failure branches; `next_index` floored at 1; do NOT recompute or advance any commit index from `match_index`
|
||||
- Observable completion: 7.1 tests pass
|
||||
- _Requirements: 5.1, 5.2, 5.3, 5.4_
|
||||
- _Boundary: raft.c_
|
||||
- _Depends: 7.1_
|
||||
|
||||
## 8. Simulator backing: in-memory log + AppendEntries routing
|
||||
|
||||
- [ ] 8.1 Add an in-memory per-node log and wire the four callbacks in `raft_sim.c`
|
||||
- Add a per-`raft_sim_node_t` in-memory entry array (`{term, bytes}` per index); implement `on_persist_entry` (append/overwrite at index), `on_get_entry_term` (term or 0), `on_truncate_from` (drop index..end), and buffer `on_send_append_entries` into a new AE RPC queue
|
||||
- Observable completion: a sim node can append, read a term back, and truncate deterministically; `raft_sim_init` wires the four callbacks for every node
|
||||
- _Requirements: 8.2, 7.5_
|
||||
- _Boundary: raft_sim.c_
|
||||
- _Depends: 6.3, 7.2_
|
||||
|
||||
- [ ] 8.2 Add synchronous AppendEntries delivery honoring partitions
|
||||
- Add `raft_sim_deliver_append_entries`: for each buffered AE, call `raft_on_append_entries_request` on the (non-partitioned) destination, then route the reply back via `raft_on_append_entries_reply`; skip partitioned nodes (mirrors the existing vote-RPC routing)
|
||||
- Observable completion: driving one append + delivery replicates the entry to a connected follower; partitioned follower receives nothing
|
||||
- _Requirements: 8.1_
|
||||
- _Boundary: raft_sim.c_
|
||||
- _Depends: 8.1_
|
||||
|
||||
- [ ] 8.3 Add a Log Matching assertion helper
|
||||
- Add a helper asserting: for any two nodes and any index `i`, if both hold an entry at `i` with equal term, their prefixes through `i` are byte-identical
|
||||
- Observable completion: the helper returns pass on a converged cluster and fails on a hand-divergent one (self-test)
|
||||
- _Requirements: 8.4_
|
||||
- _Boundary: raft_sim.c_
|
||||
- _Depends: 8.1_
|
||||
|
||||
## 9. Simulator invariant tests (Log Matching, truncation, faults, G5 regression)
|
||||
|
||||
- [ ] 9.1 Replication + partition catch-up Log Matching test
|
||||
- Test: leader appends 3 entries, replicates to a quorum, assert Log Matching; then isolate a follower across more appends, heal, assert `nextIndex` walks back, the follower catches up, and Log Matching holds
|
||||
- Observable completion: test passes end-to-end via the sim
|
||||
- _Requirements: 8.1, 8.3, 8.4_
|
||||
- _Boundary: test_raft.c, raft_sim.c_
|
||||
- _Depends: 8.2, 8.3_
|
||||
|
||||
- [ ] 9.2 Divergent-suffix truncation test (G3)
|
||||
- Test: give a follower a manufactured divergent tail (same indices, different term), replicate from a higher-term leader; assert the follower truncates the divergent suffix (`on_truncate_from`) and converges to the leader's prefix; Log Matching holds
|
||||
- Observable completion: test passes; follower's post-converge prefix equals the leader's
|
||||
- _Requirements: 8.5_
|
||||
- _Boundary: test_raft.c, raft_sim.c_
|
||||
- _Depends: 9.1_
|
||||
|
||||
- [ ] 9.3 Message drop/dup/reorder convergence test
|
||||
- Test: drop, duplicate, and reorder AE requests/replies in the sim queue; assert idempotent convergence (a duplicated AE is a no-op via the already-have branch) and Log Matching after faults clear
|
||||
- Observable completion: test passes despite injected faults
|
||||
- _Requirements: 8.3_
|
||||
- _Boundary: test_raft.c, raft_sim.c_
|
||||
- _Depends: 9.1_
|
||||
|
||||
- [ ] 9.4 Chaotic-churn G5 regression test
|
||||
- Test: a stale-term node (term lagging) rejoins under a higher-term leader; assert it adopts the higher term from the AE/heartbeat path and does NOT re-elect at a reused term; Log Matching holds across the churn
|
||||
- Observable completion: test passes; the rejoining node's term equals the leader's after the first heartbeat, and no second leader appears at the reused term
|
||||
- _Requirements: 8.6_
|
||||
- _Boundary: test_raft.c, raft_sim.c_
|
||||
- _Depends: 9.1_
|
||||
|
||||
## 10. Verification
|
||||
|
||||
- [ ] 10.1 Full library suite green under ASAN
|
||||
- Run `/xce-build lib`; all existing election tests plus the new log-replication and simulator tests pass under ASAN with zero leaks
|
||||
- Observable completion: test binary exits 0; ASAN clean; no regression in the election-core tests
|
||||
- _Requirements: 1.1, 2.1, 3.2, 4.3, 5.1, 6.1, 7.1, 8.4_
|
||||
- _Boundary: src/lib/raft, src/lib/testsuite_
|
||||
- _Depends: 9.2, 9.3, 9.4_
|
||||
26
.kiro/specs/raft-sim-conformance/brief.md
Normal file
26
.kiro/specs/raft-sim-conformance/brief.md
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# Brief: raft-sim-conformance
|
||||
|
||||
## Problem
|
||||
Once the library owns the full Raft protocol (election via `raft-library`, log via `raft-log-replication`, commit safety via `raft-commit-safety`), XCE needs an **independent verification harness** that proves the consensus core conforms to `raft/raft.tla` — without a live cluster, threads, or sockets. Today `raft_sim.c` exists but only exercises the election half; there is no oracle for the four Raft safety invariants. This is the "use the verified spec to help" payoff: a deterministic regression oracle that would have caught **bug-851** before it shipped.
|
||||
|
||||
## Approach
|
||||
Extend `src/lib/raft/raft_sim.c` into a **conformance harness** that drives N library instances purely in-process through transitions equivalent to the `raft.tla` `Next` actions, and continuously checks the TLA+ safety invariants.
|
||||
|
||||
### TLA+ actions to drive (raft.tla `Next`, line 454)
|
||||
`Restart`, `Timeout`, `RequestVote`, `BecomeLeader`, `ClientRequest`, `AppendEntries`, `AdvanceCommitIndex`, `Receive` (vote/AE request+response, `UpdateTerm`, drop-stale), `DuplicateMessage`, `DropMessage`. The harness owns a message bag and per-node state mirroring `raft.tla`'s variables.
|
||||
|
||||
### Invariants asserted every step (the four Raft safety properties)
|
||||
1. **Election Safety** — at most one leader per term.
|
||||
2. **Log Matching** — if two logs share an entry at `(index, term)`, all prior entries are identical.
|
||||
3. **Leader Completeness** — an entry committed in term T is present in every leader's log for terms > T.
|
||||
4. **State Machine Safety** — no two nodes apply different commands at the same index.
|
||||
|
||||
## Scope
|
||||
- **In**: a deterministic, seedable driver (randomized + targeted scenarios: partitions, restarts, message drop/dup/reorder, Figure-8); per-step invariant checks; failure reproduction (print the action trace that broke an invariant); CI-runnable test target alongside the existing library tests. Optionally: emit a trace consumable by TLC, or compare against TLC-generated traces of `raft.tla`.
|
||||
- **Out**: the protocol implementation itself (owned by the three upstream specs); the live controller / `/xce-deploy-ha` integration (that is `ha-term-safety`'s chaotic-failover regression); config markers/snapshot.
|
||||
|
||||
## Verification
|
||||
The harness IS the verification. Acceptance: it reproduces a bug-851-class divergence on a deliberately-broken build (e.g. term stamping disabled, or `(term,lsn)` truncation removed) and passes on the correct build; it runs M randomized seeds in CI with zero invariant violations.
|
||||
|
||||
## Dependencies
|
||||
`raft-log-replication`, `raft-commit-safety` (HARD — the harness checks invariants over the log + commit behaviour they add). Complements `ha-term-safety`'s live regression (this is the in-process oracle; that is the end-to-end one).
|
||||
584
.kiro/specs/raft-sim-conformance/design.md
Normal file
584
.kiro/specs/raft-sim-conformance/design.md
Normal file
|
|
@ -0,0 +1,584 @@
|
|||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose**: This feature turns `src/lib/raft/raft_sim.c` (today an election-only RPC
|
||||
router) into an **in-process TLA+ conformance harness** for the XCE Raft core. It drives
|
||||
N library instances synchronously through transitions equivalent to the `raft.tla`
|
||||
`Next` actions (`raft.tla:454`) and asserts the four Raft safety invariants after every
|
||||
step. It delivers an independent regression oracle for the consensus core — the "use the
|
||||
verified spec to help" payoff — that would have caught **bug-851** (the silent
|
||||
committed-prefix divergence) before it shipped.
|
||||
|
||||
**Users**: Raft library maintainers (verifying `raft.c`), the `raft-log-replication` and
|
||||
`raft-commit-safety` authors (whose entry points this harness exercises), and CI (which
|
||||
runs the harness as a first-class library test target).
|
||||
|
||||
**Impact**: Replaces ad-hoc election-only simulation with a complete in-process model of
|
||||
the `raft.tla` state space: a shared message bag mirroring `messages`, per-node state
|
||||
mirroring `serverVars`/`logVars`/`leaderVars`, an action scheduler, four invariant
|
||||
checkers, and a failure-trace recorder. The protocol implementation is unchanged — the
|
||||
harness drives it only through the published entry points and callbacks from the two
|
||||
upstream specs; the harness itself **is** the in-memory adapter (it implements the WAL/
|
||||
transport callbacks against in-memory structures).
|
||||
|
||||
### Goals
|
||||
|
||||
- A deterministic, seedable driver that mirrors the `raft.tla` `messages` bag and the
|
||||
per-node variables this harness tracks, driving the library entry points only.
|
||||
- Per-step assertion of all four safety invariants (Election Safety, Log Matching, Leader
|
||||
Completeness, State Machine Safety) computed purely from observable node state.
|
||||
- Targeted adversarial scenarios (partition/heal, restart, drop/dup/reorder, Figure-8,
|
||||
bug-851 G5 term-reuse) plus randomized seed sweeps.
|
||||
- A replayable failure trace (seed + ordered action list) and a CI-runnable target.
|
||||
- A deliberately-broken-build negative test proving the oracle catches a bug-851-class
|
||||
divergence and passes on the correct build.
|
||||
- Optionally, a TLC-consumable trace export.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- **No protocol semantics.** logOk, conflict truncation, the current-term commit rule,
|
||||
`(lastLogTerm, lastLogIndex)` freshness, and `UpdateTerm` adoption live in `raft.c`
|
||||
(owned by `raft-log-replication` / `raft-commit-safety`). The harness drives and checks
|
||||
them; it never re-implements them.
|
||||
- **No live cluster / controller integration.** The `/xce-deploy-ha` chaotic-failover
|
||||
regression, controller WAL/repl/recovery/commit-barrier internals, config markers, and
|
||||
the upgrade snapshot are `ha-term-safety`.
|
||||
- **No changes to `raft/raft.tla`** and **no TLC model-checker harness** for the spec
|
||||
(only an *optional export* the maintainer can feed to TLC manually).
|
||||
- **No async I/O, threads, or sockets.** The harness is synchronous and deterministic by
|
||||
construction.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- `src/lib/raft/raft_sim.c` — the harness body: the shared message bag, per-node
|
||||
in-memory log + commit + applied state, the in-memory adapter callbacks, the seeded
|
||||
action scheduler, the four invariant checkers, the trace recorder, the scenario
|
||||
generators, and the deliberately-broken-build fault switch wiring (test-only).
|
||||
- The harness public surface in `raft.h`'s `XCE_TESTING` section: new sim types
|
||||
(message-bag entry, per-node log array, action record) and the new `raft_sim_*` driver
|
||||
functions. These extend — never alter — the protocol declarations.
|
||||
- The conformance test entry point + scenarios in `src/lib/testsuite/test_raft.c`
|
||||
(`XCE_TESTING`-gated), including the negative (broken-build) test.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- The protocol entry points and callback **semantics** (`raft_log_append`,
|
||||
`raft_send_append_entries`, `raft_on_append_entries_request/reply`,
|
||||
`raft_advance_commit`, the vote path, `on_persist_entry`, `on_get_entry_term`,
|
||||
`on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) — owned by
|
||||
`raft-log-replication` / `raft-commit-safety`. This spec consumes them verbatim.
|
||||
- Any `raft.c` change to protocol logic. The harness must not need one; if it does, that
|
||||
is an upstream-spec gap, not a harness change.
|
||||
- The controller-side adapters, WAL byte layout, repl transport, recovery, commit
|
||||
barrier, config markers, snapshot — `ha-term-safety`.
|
||||
- `raft/raft.tla` itself and any TLC tooling.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
- The `raft-log-replication` shared callback contract **verbatim**: `on_persist_entry`,
|
||||
`on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`; the 1-based index =
|
||||
WAL LSN identity; the `term 0` = absent convention; per-peer `next_index[]`/
|
||||
`match_index[]`; entry points `raft_log_append`, `raft_send_append_entries`,
|
||||
`raft_on_append_entries_request`, `raft_on_append_entries_reply`,
|
||||
`raft_append_result_t`.
|
||||
- The `raft-commit-safety` additions: `raft_advance_commit` (current-term-gated
|
||||
`AdvanceCommitIndex`), `on_commit_advanced`, and the `(lastLogTerm, lastLogIndex)` vote
|
||||
freshness already enforced inside the vote handlers.
|
||||
- The implemented election core (`raft-library`): `raft_init`, `raft_start_election`,
|
||||
`raft_on_vote_request/reply`, `raft_on_election_timeout`, PreVote, `raft_snapshot`.
|
||||
- Only `errno.h`, `stdint.h`, `string.h`, `stdio.h`, `unity/unity.h` (test) — the
|
||||
simulator stays free-standing, like the rest of the library, and is `XCE_TESTING`-gated.
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
Re-check this design if any upstream changes:
|
||||
|
||||
- The signature/semantics of any of the four log-replication callbacks or
|
||||
`on_commit_advanced`.
|
||||
- The signature/return semantics of `raft_log_append`, `raft_send_append_entries`,
|
||||
`raft_on_append_entries_request`, `raft_on_append_entries_reply`, `raft_advance_commit`,
|
||||
or `raft_append_result_t`.
|
||||
- The 1-based-index ↔ WAL-LSN identity or the `term 0` = absent convention.
|
||||
- The placement of `UpdateTerm` / `DropStaleResponse` inside the handlers (the harness
|
||||
relies on the handlers performing these, not the harness).
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
`raft_sim.c` already establishes the harness idiom this design extends:
|
||||
|
||||
- A `raft_sim_cluster_t` owning `nodes[RAFT_SIM_MAX_NODES]`, a `pending[]` RPC queue, and
|
||||
cluster-level counters.
|
||||
- Each `raft_sim_node_t` carries `raft_state_t state`, a `raft_callbacks_t cbs` wired to
|
||||
sim routing functions, a back-pointer `ctx` to the cluster, an `idx`, a `partition`
|
||||
flag, and a `trace_buf`.
|
||||
- Callbacks cast `ctx` back to `raft_sim_node_t*`, reaching the cluster via `node->ctx`.
|
||||
- Routing is synchronous: `raft_sim_deliver_pending` drains the queue, calling
|
||||
`raft_on_vote_request` on the destination and routing the reply back.
|
||||
|
||||
This design keeps that idiom and adds: a richer message bag (vote **and** AppendEntries,
|
||||
requests **and** responses), an in-memory log + commit + applied state per node, the four
|
||||
new adapter callbacks, a seeded action scheduler, invariant checkers, and a trace
|
||||
recorder. No new file is introduced; the existing module is extended (consistent with how
|
||||
`raft-log-replication` extends `raft_sim.c`).
|
||||
|
||||
### TLA+ variable → harness state mapping
|
||||
|
||||
The harness mirrors exactly the `raft.tla` variables it needs to drive actions and check
|
||||
invariants. The protocol-internal portions of `serverVars`/`leaderVars` (`currentTerm`,
|
||||
`votedFor`, `state`, `nextIndex`, `matchIndex`) live inside each `raft_state_t` and are
|
||||
**read** through `raft_snapshot` and the library's own fields — the harness does not
|
||||
duplicate them.
|
||||
|
||||
| `raft.tla` variable | Harness home | Read/owned |
|
||||
|---|---|---|
|
||||
| `messages` (bag of in-flight RPCs) | `cluster.bag[]` (requests + responses) | harness-owned |
|
||||
| `currentTerm[i]`, `state[i]`, `votedFor[i]` | `node.state` (`raft_state_t`) | library-owned, read via snapshot/fields |
|
||||
| `log[i]` (sequence of `(index, term, value)`) | `node.log[]` in-memory array | harness-owned (the adapter) |
|
||||
| `commitIndex[i]` | `node.commit_index` | harness-owned, advanced via `raft_advance_commit` + `on_commit_advanced` |
|
||||
| `nextIndex[i][j]`, `matchIndex[i][j]` | `node.state.next_index[]/match_index[]` | library-owned |
|
||||
| applied state machine (for SM Safety) | `node.applied[]` (value per committed index) | harness-owned |
|
||||
| `allLogs` (history of every log) | `cluster.all_logs` snapshot used by checkers | harness-owned |
|
||||
|
||||
### Component / data-flow map
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph SCHED["Action Scheduler (raft_sim.c) — seeded PRNG"]
|
||||
PICK["pick action ~ raft.tla Next (454)"]
|
||||
REC["record action -> trace buffer (R1.4)"]
|
||||
end
|
||||
subgraph NODES["N library instances (raft.c — UNCHANGED)"]
|
||||
ST["raft_state_t (term, role, votedFor, next/match)"]
|
||||
EP["entry points: start_election, on_vote_*,<br/>log_append, send_append_entries,<br/>on_append_entries_*, advance_commit"]
|
||||
end
|
||||
subgraph ADAPT["In-memory adapter callbacks (raft_sim.c = the adapter)"]
|
||||
A1["on_persist_entry -> node.log[]"]
|
||||
A2["on_get_entry_term -> node.log[idx].term | 0"]
|
||||
A3["on_send_append_entries -> bag.push(AE req)"]
|
||||
A4["on_truncate_from -> node.log truncate"]
|
||||
A5["on_commit_advanced -> node.commit_index + apply()"]
|
||||
A6["on_send_vote_request -> bag.push(vote req)"]
|
||||
end
|
||||
subgraph BAG["Message bag (cluster.bag[]) — mirrors raft.tla messages"]
|
||||
B["vote/AE requests + responses;<br/>Drop/Duplicate/Reorder live here"]
|
||||
end
|
||||
subgraph CHK["Invariant checkers (after every step, R3)"]
|
||||
I1["Election Safety"]
|
||||
I2["Log Matching"]
|
||||
I3["Leader Completeness"]
|
||||
I4["State Machine Safety"]
|
||||
end
|
||||
PICK --> REC --> EP
|
||||
EP --> A1 & A2 & A3 & A4 & A5 & A6
|
||||
A3 & A6 --> BAG
|
||||
BAG -->|Receive/Deliver| EP
|
||||
EP --> CHK
|
||||
CHK -->|violation| FAIL["halt + dump seed & trace (R5)"]
|
||||
```
|
||||
|
||||
**Architecture integration**
|
||||
|
||||
- **Pattern**: model-based test harness over a pure state machine. The library is the
|
||||
system-under-test (Ports-and-Adapters: the harness implements all callback ports
|
||||
against in-memory structures). The scheduler is a seeded interleaving engine over the
|
||||
`raft.tla` action set; the checkers are pure observers.
|
||||
- **Domain boundary**: protocol decisions stay in `raft.c`; *driving* (which action,
|
||||
which message) and *checking* (the four invariants) stay in the harness.
|
||||
- **Preserved patterns**: `ctx`-back-pointer callbacks, synchronous routing,
|
||||
`XCE_TESTING` gating, per-node trace buffers, the `raft_sim_*` naming.
|
||||
- **Steering compliance**: `src/lib/CLAUDE.md` (free-standing gnu17, tests in
|
||||
`testsuite/`), `ha-async-io.md` (no blocking — and here, no async at all),
|
||||
`testing.md` (TDD: checkers/scenarios written before they pass), "controller never
|
||||
self-terminates" (the harness `abort()`s a test on violation, never the library).
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice | Role | Notes |
|
||||
|---|---|---|---|
|
||||
| Harness core | C (gnu17), `string.h`/`stdint.h`/`stdio.h` | message bag, scheduler, checkers, trace | `XCE_TESTING`-gated; free-standing |
|
||||
| PRNG | small inlined deterministic generator (e.g. SplitMix64/xorshift) seeded by a `uint64_t` | all nondeterministic choices | no libc `rand()` (not reproducible across platforms) |
|
||||
| Adapter | in-memory `node.log[]`, `commit_index`, `applied[]` | implements the 6 callbacks | the harness IS the WAL/transport adapter |
|
||||
| Test runner | Unity (`unity/unity.h`) in `test_raft.c` | scenario + seed-sweep + negative test | built by `src/lib/meson.build` test target |
|
||||
| Fault switch | compile-time macro (e.g. `RAFT_SIM_BREAK_*`) in the test build only | negative test | never set in default CI run |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Modified Files
|
||||
|
||||
- `src/lib/raft/raft.h` (`XCE_TESTING` section only) — extend the sim types: a unified
|
||||
message-bag entry (`raft_sim_msg_t`: kind = vote-req/vote-rsp/ae-req/ae-rsp, src, dst,
|
||||
and a union of payload fields incl. term, prev_log_index/term, leader_commit, entry
|
||||
bytes + entry_term, match_index, granted/success); a per-node in-memory log entry
|
||||
(`raft_sim_log_entry_t { uint64_t term; uint64_t value; }`); an action record
|
||||
(`raft_sim_action_t`: kind + actor + target/message identity); extend
|
||||
`raft_sim_node_t` with `log[]`, `log_len`, `commit_index`, `applied[]`; extend
|
||||
`raft_sim_cluster_t` with the bag, the PRNG state, the trace buffer, and the seed;
|
||||
declare the new `raft_sim_*` driver/scenario/check functions. **No protocol
|
||||
declaration is changed.**
|
||||
- `src/lib/raft/raft_sim.c` — the harness implementation (sections below): adapter
|
||||
callbacks, bag operations (push/deliver/drop/duplicate/reorder), the seeded action
|
||||
scheduler (`raft_sim_step` / `raft_sim_run`), the four invariant checkers
|
||||
(`raft_sim_check_*` + `raft_sim_check_all`), the trace recorder + dumper, the targeted
|
||||
scenario drivers, and the fault-switch hooks (compile-time-gated).
|
||||
- `src/lib/testsuite/test_raft.c` — the conformance test cases: targeted scenarios,
|
||||
the randomized M-seed sweep, the replay-reproduction test, and the negative
|
||||
(broken-build) test.
|
||||
|
||||
### Directory Structure (unchanged)
|
||||
|
||||
```
|
||||
src/lib/raft/
|
||||
├── raft.h # + sim message bag, in-memory log, action record, driver decls (XCE_TESTING)
|
||||
├── raft.c # UNCHANGED by this spec (protocol owned upstream)
|
||||
└── raft_sim.c # + bag, scheduler, 4 invariant checkers, trace, scenarios, fault hooks
|
||||
src/lib/testsuite/
|
||||
└── test_raft.c # + conformance scenarios, M-seed sweep, replay test, negative test
|
||||
```
|
||||
|
||||
> Single-responsibility kept: `raft.c` = pure protocol (untouched), `raft_sim.c` =
|
||||
> deterministic harness + in-memory adapter + checkers, `test_raft.c` = assertions and
|
||||
> the CI batch.
|
||||
|
||||
## System Flows
|
||||
|
||||
### One scheduler step (the `Next` interleaving)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant S as Scheduler (seeded PRNG)
|
||||
participant L as Library node (raft.c)
|
||||
participant A as Adapter cb (raft_sim.c)
|
||||
participant B as Message bag
|
||||
participant C as Checkers
|
||||
S->>S: action = pick(Next set) from PRNG
|
||||
S->>S: record(action) into trace (R1.4)
|
||||
alt node-local (Timeout / RequestVote / BecomeLeader / ClientRequest)
|
||||
S->>L: call entry point (start_election / log_append / ...)
|
||||
L->>A: on_persist_entry / on_send_* (drives bag/log)
|
||||
A->>B: push request(s)
|
||||
else AppendEntries (leader replicate)
|
||||
S->>L: raft_send_append_entries(peer, leader_commit)
|
||||
L->>A: on_send_append_entries -> bag.push(AE req)
|
||||
else AdvanceCommitIndex
|
||||
S->>L: raft_advance_commit()
|
||||
L->>A: on_commit_advanced -> commit_index += ; apply()
|
||||
else Receive (deliver one bag message)
|
||||
S->>B: pop selected message
|
||||
S->>L: route to handler (on_vote_request / on_append_entries_request / *_reply)
|
||||
L->>A: UpdateTerm/persist/truncate/append + reply push
|
||||
else DropMessage / DuplicateMessage
|
||||
S->>B: discard / re-enqueue (no node mutation) (R2.5)
|
||||
else Restart
|
||||
S->>L: raft_init preserving persisted term/votedFor + restore log_len (R2.1)
|
||||
end
|
||||
S->>C: raft_sim_check_all() (R3)
|
||||
alt any invariant violated
|
||||
C-->>S: fail -> dump seed + trace + invariant (R5); halt
|
||||
else ok
|
||||
C-->>S: continue
|
||||
end
|
||||
```
|
||||
|
||||
Flow-level decisions:
|
||||
|
||||
- **The library performs the protocol; the harness performs the interleaving.**
|
||||
`UpdateTerm` and `DropStaleResponse` happen *inside* the handlers (R2.4); the harness
|
||||
never pre-filters by term — that would mask the very bugs it must catch.
|
||||
- **Commit is library-gated.** The harness calls `raft_advance_commit`; the current-term
|
||||
Figure-8 rule is enforced by `raft.c`. The harness only *applies* whatever the library
|
||||
reports committed via `on_commit_advanced`, then checks State Machine Safety.
|
||||
- **Faults are bag-only.** Drop/Duplicate/Reorder mutate `cluster.bag[]` exclusively
|
||||
(R2.5), mirroring `raft.tla:443`/`448` (`UNCHANGED` of all non-message vars).
|
||||
- **Restart preserves persistence.** The harness keeps `node.log[]`, the recovered
|
||||
`log_len`, and the last-persisted `(term, votedFor)` across a `raft_init`, matching
|
||||
`raft.tla` `Restart` (keeps `currentTerm`, `votedFor`, `log`; resets volatile state).
|
||||
|
||||
### Invariant checker construction (observable-state-only)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph IN["inputs: per-node observable state"]
|
||||
R["roles+terms (raft_snapshot)"]
|
||||
LG["node.log[] (index, term, value)"]
|
||||
CM["node.commit_index"]
|
||||
AP["node.applied[]"]
|
||||
end
|
||||
R --> ES["Election Safety: |{i: role==LEADER, term==t}| <= 1, all t (R3.1)"]
|
||||
LG --> LM["Log Matching: pairwise, shared (idx,term) => identical prefix (R3.2)"]
|
||||
LG --> LC["Leader Completeness: committed entry in every higher-term leader log (R3.3)"]
|
||||
CM --> LC
|
||||
AP --> SM["State Machine Safety: applied[i][k] == applied[j][k] for all k (R3.4)"]
|
||||
```
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Design elements | raft.tla anchor |
|
||||
|---|---|---|---|
|
||||
| 1.1–1.6 | Deterministic seedable driver | seeded PRNG; `cluster.bag[]`; per-node `log[]`/`commit_index`/`applied[]`; trace buffer; no threads/sockets/clock; deterministic replay | `messages`, `serverVars`, `logVars` |
|
||||
| 2.1–2.6 | Drive Next-equivalent actions via library entry points | scheduler step dispatch; `raft_send_append_entries`; `raft_advance_commit`; handler routing (UpdateTerm/DropStale inside lib); bag-only Drop/Dup; entry-points-only rule | `Next` (454), `Restart`/`Timeout`/`RequestVote`/`BecomeLeader`/`ClientRequest`/`AppendEntries`/`AdvanceCommitIndex`/`Receive`/`Duplicate`/`Drop` |
|
||||
| 3.1–3.6 | Per-step four-invariant assertion | `raft_sim_check_election_safety/log_matching/leader_completeness/state_machine_safety`; `raft_sim_check_all`; halt-on-fail; observable-state-only | Election Safety / Log Matching / Leader Completeness / State Machine Safety |
|
||||
| 4.1–4.5 | Targeted scenarios | `raft_sim_scenario_partition_heal/restart_churn/drop_dup_reorder/figure8/stale_rejoin` | partition / Restart / message faults / Figure-8 / G5 |
|
||||
| 5.1–5.4 | Failure-trace capture + replay | trace recorder; `raft_sim_dump_trace` (seed + ordered actions); replay from seed; quiet-on-success | (verification) |
|
||||
| 6.1–6.4 | CI-runnable target | `test_raft.c` conformance entry; M-seed batch + scenarios; ASAN/UBSAN clean; bounded steps/seeds | (verification) |
|
||||
| 7.1–7.4 | Deliberately-broken-build negative test | compile-time `RAFT_SIM_BREAK_*` fault switch (test build only); both-direction assertion | bug-851 G1/G3 mechanism |
|
||||
| 8.1–8.3 | Optional TLC-consumable trace export | stable machine-readable record named per `raft.tla`; off by default | `raft.tla` actions/vars |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Layer | Intent | Req coverage | Key dependencies | Contract |
|
||||
|---|---|---|---|---|---|
|
||||
| Message bag | Harness | Mirror `messages`; carry vote/AE req+rsp | 1, 2 | `raft_sim_msg_t` | State |
|
||||
| In-memory adapter callbacks | Harness | Implement the 6 ports against `node.log[]`/commit | 1, 2 | upstream callback contract (P0) | Service |
|
||||
| Action scheduler | Harness | Seeded interleaving over the Next set | 1, 2 | PRNG, entry points (P0) | Service |
|
||||
| Invariant checkers | Harness | Assert the four safety properties | 3 | observable node state | Service |
|
||||
| Trace recorder | Harness | Record + dump replayable action trace | 1, 5 | scheduler | Service |
|
||||
| Scenario drivers | Harness | Targeted adversarial cases | 4 | scheduler, bag | Service |
|
||||
| Fault switch | Test build | Deliberately-broken build | 7 | compile-time macro | Service |
|
||||
| Conformance test | Test suite | CI batch + negative test | 6, 7 | all of the above | Test |
|
||||
|
||||
### In-memory adapter callbacks (the harness as adapter)
|
||||
|
||||
The harness implements the upstream callbacks against per-node in-memory state, exactly
|
||||
as the `raft-log-replication` design anticipates ("bytes live in the adapter (WAL) /
|
||||
simulator (in-memory array)"). Signatures are consumed verbatim from upstream; behavior:
|
||||
|
||||
| Callback | Harness behavior |
|
||||
|---|---|
|
||||
| `on_persist_entry(ctx, index, term, entry, len)` | Write/overwrite `node.log[index-1] = {term, value}`; grow `log_len`. Persist-before-act preserved (called before the library reports success). |
|
||||
| `on_get_entry_term(ctx, index)` | Return `node.log[index-1].term`, or `0` if `index==0` or `index > log_len` (the `term 0` = absent convention). |
|
||||
| `on_send_append_entries(ctx, peer, prev_log_index, prev_log_term, leader_commit, ...)` | Push an AE-request `raft_sim_msg_t` onto `cluster.bag[]` honoring partitions. |
|
||||
| `on_truncate_from(ctx, index)` | Set `node.log_len = index - 1` (delete `index..end`). |
|
||||
| `on_commit_advanced(ctx, commit_index)` | Set `node.commit_index = commit_index`; apply `node.log[k].value` into `node.applied[k]` for each newly committed `k`. |
|
||||
| `on_send_vote_request(ctx, peer, term)` (existing) | Push a vote-request message onto the bag. |
|
||||
|
||||
### Message bag (mirrors `raft.tla` `messages`)
|
||||
|
||||
```c
|
||||
typedef enum {
|
||||
RAFT_SIM_MSG_VOTE_REQ, RAFT_SIM_MSG_VOTE_RSP,
|
||||
RAFT_SIM_MSG_AE_REQ, RAFT_SIM_MSG_AE_RSP,
|
||||
} raft_sim_msg_kind_t;
|
||||
|
||||
typedef struct {
|
||||
raft_sim_msg_kind_t kind;
|
||||
int src, dst;
|
||||
uint64_t term; /* RPC term (all kinds) */
|
||||
/* AE request */
|
||||
uint64_t prev_log_index, prev_log_term, leader_commit;
|
||||
uint64_t entry_term, entry_value; int entries_len; /* 0=heartbeat, 1=one entry */
|
||||
/* responses */
|
||||
int granted; /* vote rsp */
|
||||
int success; /* AE rsp */
|
||||
uint64_t match_index; /* AE rsp */
|
||||
} raft_sim_msg_t;
|
||||
```
|
||||
|
||||
- `raft_sim_bag_push` / `raft_sim_bag_deliver(i)` (route message `i` to the matching
|
||||
handler and push any reply) / `raft_sim_bag_drop(i)` / `raft_sim_bag_duplicate(i)` /
|
||||
reorder (deliver in PRNG-chosen order). Delivery honors `node.partition` (a partitioned
|
||||
endpoint's messages are not delivered, mirroring a dropped network edge).
|
||||
- A handler reply is itself pushed onto the bag as a `*_RSP` message, so responses are
|
||||
first-class bag members (drop/dup/reorder apply to them too).
|
||||
|
||||
### Action scheduler
|
||||
|
||||
```c
|
||||
/* One step of the Next interleaving. Picks an enabled action from the PRNG,
|
||||
* records it, applies it through the library entry points / bag, then runs
|
||||
* raft_sim_check_all. Returns 0 on success, <0 (and dumps the trace) on the
|
||||
* first invariant violation. */
|
||||
int raft_sim_step(raft_sim_cluster_t *c);
|
||||
|
||||
/* Run up to max_steps (or until quiescent). Returns 0 if no invariant was
|
||||
* violated through the whole run; <0 on the first violation. */
|
||||
int raft_sim_run(raft_sim_cluster_t *c, int max_steps);
|
||||
|
||||
/* Seeded init: builds N nodes wired to the in-memory adapter + bag. */
|
||||
int raft_sim_conformance_init(raft_sim_cluster_t *c, int n_nodes,
|
||||
uint32_t election_timeout_ms, uint64_t seed);
|
||||
```
|
||||
|
||||
The action set the scheduler chooses from (each maps to a `raft.tla` `Next` disjunct):
|
||||
`Restart`, `Timeout`, `RequestVote`(deliver vote-req), `BecomeLeader`(deliver enough
|
||||
vote-rsp), `ClientRequest`, `AppendEntries`, `AdvanceCommitIndex`, `Receive`(deliver an
|
||||
AE req/rsp incl. `UpdateTerm`/`DropStaleResponse`), `DuplicateMessage`, `DropMessage`.
|
||||
Only *enabled* actions are eligible (e.g. `ClientRequest`/`AppendEntries` require a
|
||||
leader; `Receive` requires a non-empty bag). Note that `BecomeLeader` is **not** a
|
||||
standalone library entry point: it is realized by delivering the quorum-completing
|
||||
vote-response to `raft_on_vote_reply`, which returns `RAFT_BECAME_LEADER` when that
|
||||
response carries the candidate over quorum.
|
||||
|
||||
### Invariant checkers
|
||||
|
||||
```c
|
||||
int raft_sim_check_election_safety(const raft_sim_cluster_t *c); /* R3.1 */
|
||||
int raft_sim_check_log_matching(const raft_sim_cluster_t *c); /* R3.2 */
|
||||
int raft_sim_check_leader_completeness(const raft_sim_cluster_t *c); /* R3.3 */
|
||||
int raft_sim_check_state_machine_safety(const raft_sim_cluster_t *c); /* R3.4 */
|
||||
int raft_sim_check_all(const raft_sim_cluster_t *c); /* R3.5: first failure */
|
||||
```
|
||||
|
||||
Checker semantics (observable-state-only, R3.6):
|
||||
|
||||
- **Election Safety** — for each term `t`, count nodes with `role==LEADER && term==t`;
|
||||
fail if any count > 1.
|
||||
- **Log Matching** — for every node pair `(i, j)` and every index `k` present in both:
|
||||
if `log_i[k].term == log_j[k].term`, then every entry at indices `1..k` must be equal
|
||||
`(term, value)` in both logs; fail otherwise. Note: the library collapses `raft.tla`'s
|
||||
truncate-only conflict step and the immediately-following no-conflict append into a
|
||||
single AppendEntries handler invocation (truncate **then** append), so the
|
||||
Log-Matching / divergent-suffix checker authors should expect the divergent suffix to
|
||||
be truncated *and* replaced with the leader's entry in one step rather than over two.
|
||||
- **Leader Completeness** — track the highest committed entry observed cluster-wide; for
|
||||
every node currently `LEADER` (and across forced leader changes, the highest-term
|
||||
leader to date), every committed `(index, term)` must be present in its log; fail
|
||||
otherwise. (Committedness is taken from `node.commit_index` produced by the
|
||||
library-gated `raft_advance_commit`.)
|
||||
- **State Machine Safety** — for every index `k` applied by two nodes, `applied_i[k] ==
|
||||
applied_j[k]`; fail otherwise.
|
||||
|
||||
### Trace recorder
|
||||
|
||||
`raft_sim_action_t { kind, actor, msg_index, term, index }` recorded per step into
|
||||
`cluster.trace[]`; `raft_sim_dump_trace(c, fp)` prints `seed`, `n`, the failing invariant,
|
||||
and the ordered actions in a human-readable form (R5.1, R5.2). Replay = re-run
|
||||
`raft_sim_conformance_init` with the same `(n, seed)` and the same `max_steps` (R5.3);
|
||||
quiet on success unless `RAFT_SIM_VERBOSE` (R5.4).
|
||||
|
||||
### Deliberately-broken-build fault switch (test-only)
|
||||
|
||||
Compile-time macros gate **harness-side adapter** corruptions that emulate the bug-851
|
||||
defects without touching `raft.c`:
|
||||
|
||||
- `RAFT_SIM_BREAK_TERM_STAMP` — `on_persist_entry` writes `term = 0` instead of the
|
||||
supplied term (emulates G1: entry term always 0), so `on_get_entry_term` can no longer
|
||||
distinguish `(term=A, idx=k)` from `(term=B, idx=k)`.
|
||||
- `RAFT_SIM_BREAK_TRUNCATE` — `on_truncate_from` is a no-op (emulates G3: the divergent
|
||||
suffix is never deleted).
|
||||
|
||||
Under either macro the targeted scenarios (Figure-8, stale-rejoin, divergent-suffix) must
|
||||
trip Log Matching / Leader Completeness / State Machine Safety; with the macros off, the
|
||||
same run is clean. This keeps the negative test entirely inside the harness build
|
||||
configuration (R7.3) — the shipped protocol and the default CI run are unaffected.
|
||||
|
||||
> **Why adapter-side and not a `raft.c` edit:** the harness owns only `raft_sim.c`/tests.
|
||||
> Emulating G1 by having the *adapter* drop the term reproduces the exact observable
|
||||
> divergence (same-index/different-term entries become indistinguishable) that the real
|
||||
> bug caused, while honoring the spec boundary (no protocol modification). It still proves
|
||||
> the *checkers* have teeth: they catch the divergence regardless of where it originates.
|
||||
|
||||
### Optional TLC-consumable trace export
|
||||
|
||||
`raft_sim_export_trace(c, fp)` (off unless `RAFT_SIM_EXPORT`) emits, per step, the action
|
||||
and the resulting tracked variables (`messages` summary, per-node `currentTerm`/`state`/
|
||||
`log`/`commitIndex`) named per `raft.tla` (R8.1, R8.2), in a stable line-oriented format
|
||||
the maintainer can post-process for TLC cross-checking. Off by default (R8.3).
|
||||
|
||||
## Data Models
|
||||
|
||||
### Domain Model
|
||||
|
||||
- **Bag message** `raft_sim_msg_t` — a single in-flight RPC (vote/AE, request/response),
|
||||
the unit Drop/Duplicate/Reorder operate on.
|
||||
- **Log entry (sim)** `raft_sim_log_entry_t { uint64_t term; uint64_t value; }` — index
|
||||
is the 1-based array position (= WAL LSN); `value` is a synthetic monotone client value
|
||||
so State Machine Safety divergence is observable.
|
||||
- **Action record** `raft_sim_action_t` — `(kind, actor, msg_index, term, index)`, the
|
||||
replayable trace unit.
|
||||
|
||||
### Logical Data Model
|
||||
|
||||
- `raft_sim_node_t` gains `log[RAFT_SIM_MAX_LOG]`, `log_len`, `commit_index`,
|
||||
`applied[RAFT_SIM_MAX_LOG]`, and a persisted-`(term, votedFor)` shadow for Restart.
|
||||
- `raft_sim_cluster_t` gains `bag[RAFT_SIM_MAX_MSGS]` + `bag_count`, the PRNG state, the
|
||||
`seed`, and `trace[RAFT_SIM_MAX_STEPS]` + `trace_len`.
|
||||
- Fixed-size arrays only (no allocation), bounded by `RAFT_SIM_MAX_LOG`/`MSGS`/`STEPS`,
|
||||
keeping the harness allocation-free and ASAN-clean.
|
||||
|
||||
### Data Contracts & Integration
|
||||
|
||||
- **Entries handle**: one entry per AE (`entries_len ∈ {0,1}`), matching the upstream
|
||||
`raft.tla` "≤1 entry" simplification and the log-replication wire shape — the harness
|
||||
reuses that shape exactly.
|
||||
- **`term 0` sentinel**: honored by `on_get_entry_term` for absent indices.
|
||||
- **1-based index = LSN**: `node.log[index-1]` is the canonical mapping.
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Strategy
|
||||
|
||||
The harness is test code: on an invariant violation it dumps the seed + trace and fails
|
||||
the Unity assertion (the test aborts), **never** the library (`raft.c` returns status
|
||||
codes only — "controller never self-terminates" is preserved; the harness's halt is a
|
||||
test failure, not a library abort).
|
||||
|
||||
### Error Categories and Responses
|
||||
|
||||
- **Invariant violation** → `raft_sim_check_all` returns the failing invariant; the
|
||||
scheduler dumps `(seed, n, invariant, trace)` and returns `<0`; the test fails (R3.5,
|
||||
R5.1).
|
||||
- **Bag overflow / log overflow** (a scenario exceeded a fixed bound) → the harness
|
||||
fails loud (a distinct, non-invariant test error) rather than silently truncating, so
|
||||
bounds bugs are visible.
|
||||
- **Misuse** (e.g. `ClientRequest` driven on a non-leader) → the scheduler only selects
|
||||
*enabled* actions, so the library's `-EPERM` path is exercised intentionally (a
|
||||
targeted unit test), never as an accidental scheduler state.
|
||||
|
||||
### Monitoring
|
||||
|
||||
Reuse the existing per-node `trace_buf` and the library `on_trace` events
|
||||
(`"ae_recv"`, `"ae_conflict_truncate"`, `"term_adopt_ae"`, `"ae_reply_advance"`, etc.)
|
||||
plus the new harness-level action trace, so a failing run shows both the library's view
|
||||
and the scheduler's interleaving.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
TDD-mandatory (`testing.md`): every checker and scenario is written as a RED test in
|
||||
`test_raft.c` (with the `raft_sim.c` backing) that fails before the harness logic exists
|
||||
and passes after. All tests are `XCE_TESTING`-gated and run via `/xce-build lib`.
|
||||
|
||||
### Unit tests (checkers in isolation)
|
||||
|
||||
1. Each checker flags a hand-constructed violating cluster and passes a valid one:
|
||||
two leaders same term (Election Safety); same `(idx, term)` with different prefix
|
||||
(Log Matching); a committed entry missing from a higher-term leader (Leader
|
||||
Completeness); divergent `applied[k]` (State Machine Safety). (R3.1–3.4)
|
||||
2. `raft_sim_check_all` returns the first failing invariant and halts (R3.5).
|
||||
|
||||
### Scenario tests (targeted, R4)
|
||||
|
||||
1. Partition/heal: isolate a minority during appends, heal, drain; assert all four
|
||||
invariants hold throughout and after convergence (R4.1).
|
||||
2. Restart churn: restart nodes mid-replication preserving persisted state; assert
|
||||
re-convergence with no violation (R4.2).
|
||||
3. Drop/dup/reorder: perturb bag delivery; assert idempotent convergence + invariants
|
||||
(R4.3).
|
||||
4. Figure-8: a previous-term entry on a quorum, leader change; assert it is **not**
|
||||
reported committed until a current-term entry commits (R4.4).
|
||||
5. Stale-rejoin (bug-851 G5): a term-lagging node rejoins under a higher-term leader;
|
||||
assert it adopts the higher term from the AE/heartbeat path and does not re-elect at a
|
||||
reused term (R4.5).
|
||||
|
||||
### Randomized + CI (R6)
|
||||
|
||||
1. M-seed sweep: run `raft_sim_run` for a fixed `max_steps` across M seeds; assert zero
|
||||
violations. Bounded so the CI target fits the library test-suite budget (R6.4).
|
||||
2. Runs clean under ASAN/UBSAN (R6.3).
|
||||
|
||||
### Negative test (R7)
|
||||
|
||||
1. With `RAFT_SIM_BREAK_TERM_STAMP` (or `RAFT_SIM_BREAK_TRUNCATE`) set, the Figure-8 /
|
||||
divergent-suffix scenario **fails** an invariant and the trace is dumped; with the
|
||||
macro off, the same scenario **passes** — both directions asserted in one test, proving
|
||||
the oracle's sensitivity (R7.1, R7.2, R7.4). The macro is confined to the test build
|
||||
(R7.3).
|
||||
|
||||
### Replay test (R5)
|
||||
|
||||
1. A run that fails for seed `S` reproduces the identical failing action and invariant
|
||||
when re-run with seed `S` and the same `max_steps` (R5.3).
|
||||
163
.kiro/specs/raft-sim-conformance/requirements.md
Normal file
163
.kiro/specs/raft-sim-conformance/requirements.md
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
XCE's Raft consensus core lives in `src/lib/raft/raft.{c,h}` as a pure, callback-driven
|
||||
state machine. With `raft-log-replication` (the `(term, index)` log, `AppendEntries`,
|
||||
`HandleAppendEntriesRequest`, `UpdateTerm`-on-any-RPC) and `raft-commit-safety`
|
||||
(`AdvanceCommitIndex` with the Figure-8 current-term rule, `(lastLogTerm, lastLogIndex)`
|
||||
vote freshness) the library owns the full election + log + commit halves of
|
||||
`raft/raft.tla` (Ongaro canonical). What is still missing is an **independent oracle**:
|
||||
nothing continuously proves the implementation actually conforms to the four Raft safety
|
||||
invariants. Today `src/lib/raft/raft_sim.c` exercises only the election RPC path; it has
|
||||
no log, no commit, and no invariant checkers.
|
||||
|
||||
This feature extends `raft_sim.c` into an **in-process conformance/verification
|
||||
harness** that drives N library instances purely synchronously (no threads, no sockets,
|
||||
no live cluster) through transitions equivalent to the `raft.tla` `Next` actions
|
||||
(`raft.tla:454`) and asserts the four TLA+ safety invariants on every step. The harness
|
||||
is the "use the verified spec to help" payoff: a deterministic regression oracle for the
|
||||
consensus core. Its acceptance bar is concrete — it must **reproduce a bug-851-class
|
||||
invariant divergence on a deliberately-broken build** (e.g. term stamping disabled or
|
||||
`(term, index)` conflict truncation removed) and **pass on the correct build**, and it
|
||||
must run M randomized seeds in CI with zero invariant violations.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: a seedable/deterministic action driver mirroring `raft.tla`'s message
|
||||
bag + per-node variables; randomized and targeted scenario generators (network
|
||||
partitions, node restarts, message drop/duplicate/reorder, the Figure-8 commit
|
||||
scenario); per-step assertion of all four safety invariants over the N library
|
||||
instances; failure-trace capture and reproduction (print the exact action sequence
|
||||
that broke an invariant, replayable from the seed); a CI-runnable test target
|
||||
alongside the existing library tests; an acceptance/negative test that a
|
||||
deliberately-broken protocol build is caught and the correct build passes.
|
||||
Optionally, an exported machine-readable trace format consumable by TLC for
|
||||
cross-checking against `raft.tla`.
|
||||
- **Out of scope**: the protocol implementation itself — the log model, `AppendEntries`,
|
||||
`AdvanceCommitIndex`, vote-freshness, and term-adoption semantics are owned by
|
||||
`raft-log-replication` and `raft-commit-safety`; this harness only *drives and checks*
|
||||
them. Also out: the live controller / `/xce-deploy-ha` chaotic-failover regression
|
||||
(owned by `ha-term-safety`); the controller WAL/repl/recovery/commit-barrier
|
||||
internals; config-change markers and the upgrade snapshot (`ha-term-safety`); any
|
||||
change to `raft.tla` itself or a TLC model-checking harness for the spec.
|
||||
- **Adjacent expectations**: the harness consumes the `raft-log-replication` shared
|
||||
callback contract verbatim (`on_persist_entry`, `on_get_entry_term`,
|
||||
`on_send_append_entries`, `on_truncate_from`; 1-based index = WAL LSN; `term 0` means
|
||||
absent) and the `raft-commit-safety` additions (`on_commit_advanced`,
|
||||
`(lastLogTerm, lastLogIndex)` freshness). It re-defines none of these; if any of those
|
||||
signatures change, this spec re-validates. The simulator backing log + commit state
|
||||
lives in-memory per node (the harness is the adapter), exactly as the upstream design
|
||||
docs anticipate (`raft-log-replication` design: "bytes live in the adapter (WAL) /
|
||||
simulator (in-memory array)").
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Deterministic seedable action driver
|
||||
|
||||
**Objective:** As a Raft library maintainer, I want a fully deterministic, seed-driven
|
||||
in-process driver over N library instances, so that any invariant violation is exactly
|
||||
reproducible from its seed without threads, sockets, or a live cluster.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the harness is initialized with a node count N (1 ≤ N ≤ `RAFT_SIM_MAX_NODES`) and a 64-bit seed, the Raft Conformance Harness shall construct N library instances wired to in-memory adapter callbacks and a single shared message bag, and shall produce byte-identical execution for the same (N, seed) inputs across runs.
|
||||
2. The Raft Conformance Harness shall derive every nondeterministic choice (which action to take, which node, which message to deliver/drop/duplicate, partition toggles, restart targets, client values) from a single seeded pseudo-random source, so that no wall-clock time, address values, or uninitialized memory influence the run.
|
||||
3. While a run is in progress, the Raft Conformance Harness shall maintain per-node state mirroring the `raft.tla` variables it owns (in-memory log of `(index, term, value)` entries, `commitIndex`, applied/state-machine view) and a message bag of in-flight RPC requests and responses.
|
||||
4. When a step is executed, the Raft Conformance Harness shall record the chosen action (action kind, actor node, target/message identity) into a replayable trace buffer before applying it.
|
||||
5. The Raft Conformance Harness shall execute purely synchronously: it shall not spawn threads, open sockets, perform real I/O, or read wall-clock time during a run.
|
||||
6. Where the seed and node count are fixed, when the same number of steps is executed twice, the Raft Conformance Harness shall reach an identical final per-node and message-bag state (deterministic replay).
|
||||
|
||||
### Requirement 2: Drive transitions equivalent to the raft.tla Next actions
|
||||
|
||||
**Objective:** As a Raft library maintainer, I want the harness to drive transitions
|
||||
equivalent to every `raft.tla` `Next` action through the actual library entry points, so
|
||||
that the implementation — not a re-modeled protocol — is what is being exercised.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the driver selects a node-local action, the Raft Conformance Harness shall be able to apply `Restart` (re-init a node preserving its persisted `term`/`votedFor`/log per `raft.tla` `Restart`), `Timeout`/`RequestVote` (begin an election via the election entry points), `BecomeLeader` (on a quorum of granted votes), and `ClientRequest` (append a value on a leader via `raft_log_append`).
|
||||
2. When a leader is selected for replication, the Raft Conformance Harness shall emit `AppendEntries` via the library's `raft_send_append_entries` entry point (computing `prevLogIndex`/`prevLogTerm` from the library's per-peer progress), placing the resulting RPC into the message bag.
|
||||
3. When the driver advances commit, the Raft Conformance Harness shall invoke the library's `AdvanceCommitIndex` entry point (`raft-commit-safety`) and apply the resulting committed entries to that node's in-memory state machine.
|
||||
4. When a message is delivered from the bag, the Raft Conformance Harness shall route it through the matching library handler (`raft_on_vote_request`/`raft_on_vote_reply`, `raft_on_append_entries_request`/`raft_on_append_entries_reply`), including the `UpdateTerm` higher-term adoption and the stale-response drop (`DropStaleResponse`) that those handlers implement.
|
||||
5. When a message is selected for a network fault, the Raft Conformance Harness shall apply `DuplicateMessage` (re-enqueue an in-flight message) or `DropMessage` (discard it) without otherwise mutating node state, mirroring `raft.tla:443`/`raft.tla:448`.
|
||||
6. The Raft Conformance Harness shall drive these transitions **only** through the published library entry points and callbacks; it shall not reach into or re-implement the protocol decision logic (logOk, conflict truncation, current-term commit rule, freshness) that lives in `raft.c`.
|
||||
|
||||
### Requirement 3: Per-step assertion of the four Raft safety invariants
|
||||
|
||||
**Objective:** As a Raft library maintainer, I want all four TLA+ safety invariants
|
||||
checked after every applied action, so that the first divergence is caught at the step
|
||||
that causes it, not many steps later.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. After every applied action, the Raft Conformance Harness shall assert **Election Safety**: no two distinct nodes are in the `LEADER` role for the same term.
|
||||
2. After every applied action, the Raft Conformance Harness shall assert **Log Matching**: for any two nodes whose logs contain an entry at the same index with the same term, all entries at and before that index are identical (same term and value) across those two logs.
|
||||
3. After every applied action, the Raft Conformance Harness shall assert **Leader Completeness**: every entry that has been committed by any node is present, at the same index and term, in the log of every node that is currently or subsequently a leader of a higher term.
|
||||
4. After every applied action, the Raft Conformance Harness shall assert **State Machine Safety**: no two nodes have applied different values at the same log index.
|
||||
5. If any of the four invariant checks fails, then the Raft Conformance Harness shall halt the run immediately and report the failing invariant.
|
||||
6. The Raft Conformance Harness shall compute the invariant checks only from observable per-node state (roles, terms, in-memory logs, commit indices, applied values), so that the checkers are independent of the library's internal protocol logic.
|
||||
|
||||
### Requirement 4: Targeted adversarial scenarios
|
||||
|
||||
**Objective:** As a Raft library maintainer, I want named targeted scenarios that exercise
|
||||
the exact conditions behind bug-851 and the canonical Raft hazards, so that known-hard
|
||||
cases are covered deterministically rather than left to chance.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Raft Conformance Harness shall provide a network-partition scenario that isolates a subset of nodes for a number of steps and then heals it, and shall assert all four invariants throughout and after convergence.
|
||||
2. The Raft Conformance Harness shall provide a restart scenario that restarts nodes (preserving persisted `term`/`votedFor`/log) during replication, and shall assert that restarted nodes re-converge without invariant violation.
|
||||
3. The Raft Conformance Harness shall provide a drop/duplicate/reorder scenario that perturbs the message-bag delivery order and multiplicity, and shall assert idempotent convergence with all four invariants held.
|
||||
4. The Raft Conformance Harness shall provide a Figure-8 scenario (a previous-term entry replicated to a quorum but not yet committed under the current leader) and shall assert that no node reports the previous-term entry as committed until a current-term entry is committed (Leader Completeness / State Machine Safety preserved).
|
||||
5. When a stale-term node rejoins under a higher-term leader (the bug-851 G5 mechanism), the Raft Conformance Harness shall assert that the rejoining node adopts the higher term from the AppendEntries/heartbeat path and does not re-elect at a reused term (Election Safety preserved).
|
||||
|
||||
### Requirement 5: Failure-trace capture and reproduction
|
||||
|
||||
**Objective:** As a Raft library maintainer, I want a precise, replayable failure trace
|
||||
whenever an invariant breaks, so that I can reproduce and diagnose the exact divergence.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. If an invariant check fails, then the Raft Conformance Harness shall emit the seed, the node count, the failing invariant name, and the full ordered action trace leading to the failure.
|
||||
2. The Raft Conformance Harness shall format the emitted trace so that each action is human-readable (action kind, actor node, message/target identity, relevant terms/indices).
|
||||
3. When the harness is re-run with a previously failing seed and node count, the Raft Conformance Harness shall deterministically reproduce the same failing action and invariant (replay reproducibility).
|
||||
4. When a run completes without any invariant failure, the Raft Conformance Harness shall report success quietly (no trace dump) unless verbose tracing is explicitly requested.
|
||||
|
||||
### Requirement 6: CI-runnable conformance target
|
||||
|
||||
**Objective:** As a Raft library maintainer, I want the conformance harness to run as a
|
||||
first-class test target in CI, so that consensus regressions are caught automatically
|
||||
before they ship.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The Raft Conformance Harness shall expose a test entry point in the existing library test suite (`src/lib/testsuite/`) that is built and run under the existing `XCE_TESTING` build (`/xce-build lib`).
|
||||
2. When the conformance test runs in CI, the Raft Conformance Harness shall execute a fixed batch of M randomized seeds (M ≥ a configured minimum) plus all targeted scenarios, and shall fail the test run if any seed or scenario violates any invariant.
|
||||
3. While running under sanitizers (ASAN/UBSAN as used by the library tests), the Raft Conformance Harness shall complete the CI batch without memory errors, leaks, or undefined behavior.
|
||||
4. The Raft Conformance Harness shall bound each run (a maximum step count and a fixed seed batch) so that the CI target completes within the existing library test-suite time budget.
|
||||
|
||||
### Requirement 7: Deliberately-broken-build acceptance (negative test)
|
||||
|
||||
**Objective:** As a Raft library maintainer, I want proof the oracle actually has teeth,
|
||||
so that a passing conformance run is meaningful — the harness must catch a real
|
||||
bug-851-class defect when one is present.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Where a deliberately-broken protocol build is selected by a build-time fault-injection switch (for example, term stamping disabled so entries carry term 0, or `(term, index)` conflict truncation removed), the Raft Conformance Harness shall, under that switch, reproduce a bug-851-class invariant divergence (a Log Matching, Leader Completeness, or State Machine Safety violation) within the bounded run.
|
||||
2. When the fault-injection switch is off (the correct build), the Raft Conformance Harness shall run the same scenarios and seeds with zero invariant violations.
|
||||
3. The Raft Conformance Harness shall confine the fault-injection switch to the harness/test build configuration so that it neither alters the shipped protocol implementation nor remains enabled in the default CI conformance run.
|
||||
4. The negative test shall assert both directions: the broken configuration fails (an invariant violation is detected and reported) and the correct configuration passes (no violation), so that the oracle's sensitivity is itself verified.
|
||||
|
||||
### Requirement 8: Optional TLC-consumable trace export
|
||||
|
||||
**Objective:** As a Raft library maintainer, I want the harness to optionally emit a
|
||||
machine-readable action/state trace, so that a harness run can be cross-checked against
|
||||
TLC's exploration of `raft.tla`.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Where trace export is enabled, the Raft Conformance Harness shall emit each step as a machine-readable record (action and the resulting values of the `raft.tla` variables the harness tracks) in a stable, documented format.
|
||||
2. Where trace export is enabled, the Raft Conformance Harness shall name actions and variables consistently with `raft/raft.tla` so the export can be mapped to spec actions without re-interpretation.
|
||||
3. The Raft Conformance Harness shall keep trace export off by default so that the standard CI conformance run incurs no export overhead.
|
||||
23
.kiro/specs/raft-sim-conformance/spec.json
Normal file
23
.kiro/specs/raft-sim-conformance/spec.json
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"feature_name": "raft-sim-conformance",
|
||||
"created_at": "2026-06-21T13:00:00Z",
|
||||
"updated_at": "2026-06-21T13:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true,
|
||||
"note": "Auto-approved (-y mode). Extends src/lib/raft/raft_sim.c into an in-process TLA+ conformance/verification harness driving N library instances through raft.tla Next-action-equivalent transitions (Restart, Timeout, RequestVote, BecomeLeader, ClientRequest, AppendEntries, AdvanceCommitIndex, Receive incl. UpdateTerm + DropStaleResponse, DuplicateMessage, DropMessage) and asserting the four Raft safety invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety) every step. Seedable/deterministic driver; randomized + targeted scenarios (partitions, restarts, drop/dup/reorder, Figure-8); failure-trace reproduction; CI-runnable target; deliberately-broken-build negative test reproducing a bug-851-class divergence. Boundary: owns src/lib/raft/raft_sim.c + harness/test files ONLY; does NOT modify the protocol implementation (raft.c/raft.h protocol semantics) — it consumes the raft-log-replication + raft-commit-safety entry points/callbacks verbatim. Depends: raft-log-replication, raft-commit-safety."
|
||||
}
|
||||
210
.kiro/specs/raft-sim-conformance/tasks.md
Normal file
210
.kiro/specs/raft-sim-conformance/tasks.md
Normal file
|
|
@ -0,0 +1,210 @@
|
|||
# Implementation Plan
|
||||
|
||||
> **TDD-mandatory** (`testing.md`): every checker, scheduler action, and scenario lands as
|
||||
> a RED test in `src/lib/testsuite/test_raft.c` (with the `raft_sim.c` backing) that fails
|
||||
> before the harness logic exists and passes after. Build/run library tests via
|
||||
> `/xce-build lib`. All harness code is `XCE_TESTING`-gated.
|
||||
>
|
||||
> **Spec boundary** (applies to ALL tasks): this spec owns `src/lib/raft/raft_sim.c`, the
|
||||
> `XCE_TESTING` sim section of `src/lib/raft/raft.h`, and the conformance tests in
|
||||
> `src/lib/testsuite/test_raft.c` ONLY. It does **NOT** modify the protocol
|
||||
> implementation: no task changes `raft.c` protocol logic or any protocol declaration in
|
||||
> `raft.h`, and no task edits a controller file. The harness drives the published library
|
||||
> entry points and consumes the `raft-log-replication` + `raft-commit-safety` callback
|
||||
> contract **verbatim**.
|
||||
>
|
||||
> **Depends** (whole spec): `raft-log-replication` (log model + AppendEntries entry points
|
||||
> + the four callbacks), `raft-commit-safety` (`raft_advance_commit`, `on_commit_advanced`,
|
||||
> `(lastLogTerm, lastLogIndex)` freshness). If those signatures change, re-validate per the
|
||||
> design's Revalidation Triggers.
|
||||
|
||||
## 1. Harness types and in-memory adapter backing
|
||||
|
||||
- [ ] 1.1 Extend the `XCE_TESTING` sim section of `raft.h` with the harness data types
|
||||
- Add `raft_sim_msg_kind_t`, `raft_sim_msg_t` (vote/AE req+rsp union fields), `raft_sim_log_entry_t {term, value}`, `raft_sim_action_t {kind, actor, msg_index, term, index}`, and the `RAFT_SIM_MAX_LOG/MSGS/STEPS` bounds
|
||||
- Extend `raft_sim_node_t` with `log[]`, `log_len`, `commit_index`, `applied[]`, and a persisted-`(term, votedFor)` shadow for Restart; extend `raft_sim_cluster_t` with `bag[]`+`bag_count`, PRNG state, `seed`, and `trace[]`+`trace_len`
|
||||
- Observable completion: `raft.h` compiles under `XCE_TESTING`; no protocol declaration changed; `sizeof(raft_sim_cluster_t)` grows by the new fixed arrays
|
||||
- _Requirements: 1.1, 1.3, 2.5_
|
||||
- _Boundary: raft.h (XCE_TESTING section only)_
|
||||
|
||||
- [ ] 1.2 Implement the seeded PRNG and the in-memory adapter callbacks
|
||||
- Add a small inlined deterministic generator (e.g. SplitMix64) seeded by `cluster.seed`; add `on_persist_entry` → `node.log[]`, `on_get_entry_term` → term|0, `on_truncate_from` → shrink `log_len`, `on_commit_advanced` → set `commit_index` + apply into `applied[]`; route `on_send_vote_request`/`on_send_append_entries` to a bag push
|
||||
- Add `raft_sim_conformance_init(c, n, timeout, seed)` wiring all six callbacks + the bag
|
||||
- Observable completion: a test initializes a cluster, drives one `raft_log_append` on a forced leader, and reads the entry back via the in-memory log and `on_get_entry_term`
|
||||
- _Requirements: 1.1, 1.2, 2.1_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 1.1_
|
||||
|
||||
## 2. Message bag (mirrors raft.tla `messages`)
|
||||
|
||||
- [ ] 2.1 RED+GREEN: bag push/deliver routes vote and AppendEntries RPCs through the library handlers
|
||||
- Test: pushing a vote-req and an AE-req then delivering them invokes `raft_on_vote_request` / `raft_on_append_entries_request`, and the handler reply is pushed back as a `*_RSP` message
|
||||
- Implement `raft_sim_bag_push` and `raft_sim_bag_deliver(i)` (route by `kind`, honor `node.partition`, push replies)
|
||||
- Observable completion: delivered requests produce matching responses in the bag; partitioned endpoints are not delivered
|
||||
- _Requirements: 2.2, 2.4_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 1.2_
|
||||
|
||||
- [ ] 2.2 RED+GREEN: Drop / Duplicate / Reorder mutate only the bag
|
||||
- Test: `raft_sim_bag_drop`/`_duplicate` change bag multiplicity without mutating any node's term/role/log; reorder delivers in a PRNG-chosen order
|
||||
- Implement the three bag operations (mirroring `raft.tla:443`/`448` `UNCHANGED` of non-message vars)
|
||||
- Observable completion: node snapshots are identical before/after a drop or duplicate of an undelivered message
|
||||
- _Requirements: 2.5_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 2.1_
|
||||
|
||||
## 3. The four invariant checkers (TDD against hand-built clusters)
|
||||
|
||||
- [ ] 3.1 RED+GREEN: Election Safety checker (P)
|
||||
- Test: a cluster with two `LEADER` nodes at the same term fails; a single-leader-per-term cluster passes
|
||||
- Implement `raft_sim_check_election_safety` over roles+terms read from `raft_snapshot`
|
||||
- Observable completion: both directions assert correctly
|
||||
- _Requirements: 3.1, 3.6_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 3.2 RED+GREEN: Log Matching checker (P)
|
||||
- Test: two logs sharing `(idx, term)` but differing earlier fail; identical-prefix logs pass
|
||||
- Implement `raft_sim_check_log_matching` (pairwise; shared `(idx, term)` ⇒ identical `1..idx`)
|
||||
- Observable completion: both directions assert correctly
|
||||
- _Requirements: 3.2, 3.6_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 3.3 RED+GREEN: Leader Completeness checker (P)
|
||||
- Test: a committed `(index, term)` absent from a higher-term leader's log fails; present passes
|
||||
- Implement `raft_sim_check_leader_completeness` using `commit_index` + per-node logs, tracking the highest-term leader to date
|
||||
- Observable completion: both directions assert correctly
|
||||
- _Requirements: 3.3, 3.6_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 3.4 RED+GREEN: State Machine Safety checker (P)
|
||||
- Test: divergent `applied[k]` across two nodes fails; identical applied prefixes pass
|
||||
- Implement `raft_sim_check_state_machine_safety` over `applied[]`
|
||||
- Observable completion: both directions assert correctly
|
||||
- _Requirements: 3.4, 3.6_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 3.5 RED+GREEN: `raft_sim_check_all` returns the first failing invariant and halts
|
||||
- Test: a cluster violating two invariants reports the first; a clean cluster returns OK
|
||||
- Implement `raft_sim_check_all` composing 3.1–3.4 in a fixed order
|
||||
- Observable completion: first-failure semantics verified
|
||||
- _Requirements: 3.5_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 3.1, 3.2, 3.3, 3.4_
|
||||
|
||||
## 4. Action scheduler and trace recorder
|
||||
|
||||
- [ ] 4.1 RED+GREEN: trace recorder records every action before it is applied
|
||||
- Test: after a fixed step sequence, `cluster.trace[]` holds the actions in order; `raft_sim_dump_trace` prints seed, n, and the ordered actions
|
||||
- Implement `raft_sim_action_t` recording (R1.4) + `raft_sim_dump_trace` (R5.1, R5.2); quiet on success unless `RAFT_SIM_VERBOSE` (R5.4)
|
||||
- Observable completion: a recorded trace round-trips to a readable dump
|
||||
- _Requirements: 1.4, 5.1, 5.2, 5.4_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 1.1_
|
||||
|
||||
- [ ] 4.2 RED+GREEN: `raft_sim_step` drives one Next-equivalent action through the library and checks invariants
|
||||
- Test: stepping reaches each enabled action kind over a run; every step calls `raft_sim_check_all`; a forced violation halts with a trace dump and a `<0` return
|
||||
- Implement the seeded scheduler dispatch (Restart, Timeout, RequestVote/BecomeLeader via vote delivery, ClientRequest, AppendEntries via `raft_send_append_entries`, AdvanceCommitIndex via `raft_advance_commit`, Receive via bag deliver, Duplicate/Drop) selecting only **enabled** actions, driving **only** library entry points
|
||||
- Observable completion: a multi-step run touches log append, AE replication, commit advance, and message faults; invariants checked each step
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.6, 3.5_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 2.2, 3.5, 4.1_
|
||||
|
||||
- [ ] 4.3 RED+GREEN: Restart preserves persisted `term`/`votedFor`/log
|
||||
- Test: a Restart action re-inits a node while keeping `node.log[]`, recovered `log_len`, and last-persisted `(term, votedFor)`; volatile state resets per `raft.tla` `Restart`
|
||||
- Implement the Restart branch (feed `initial_log_len` + restored `(term, votedFor)` into `raft_init`)
|
||||
- Observable completion: a restarted node retains its log and term; no invariant violation results from the restart alone
|
||||
- _Requirements: 2.1_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 4.2_
|
||||
|
||||
- [ ] 4.4 RED+GREEN: deterministic replay from `(n, seed, max_steps)`
|
||||
- Test: two runs with the same `(n, seed, max_steps)` reach identical final per-node + bag state and an identical recorded trace
|
||||
- Implement `raft_sim_run(c, max_steps)` and confirm no wall-clock/address/uninitialized input leaks into choices
|
||||
- Observable completion: byte-identical replay verified across two runs
|
||||
- _Requirements: 1.5, 1.6, 5.3_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 4.2_
|
||||
|
||||
## 5. Targeted adversarial scenarios
|
||||
|
||||
- [ ] 5.1 Partition/heal scenario (P)
|
||||
- Implement `raft_sim_scenario_partition_heal`: isolate a minority during appends, heal, drain the bag; assert all four invariants throughout and after convergence
|
||||
- Observable completion: scenario passes with zero violations on the correct build
|
||||
- _Requirements: 4.1_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 4.2_
|
||||
|
||||
- [ ] 5.2 Restart-churn scenario (P)
|
||||
- Implement `raft_sim_scenario_restart_churn`: restart nodes mid-replication preserving persisted state; assert re-convergence with no violation
|
||||
- Observable completion: scenario passes; restarted nodes converge to the leader's prefix
|
||||
- _Requirements: 4.2_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 4.3_
|
||||
|
||||
- [ ] 5.3 Drop/duplicate/reorder scenario (P)
|
||||
- Implement `raft_sim_scenario_drop_dup_reorder`: perturb bag delivery order + multiplicity; assert idempotent convergence with all four invariants held
|
||||
- Observable completion: scenario passes; duplicated AE is a no-op (already-have branch)
|
||||
- _Requirements: 4.3_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 4.2_
|
||||
|
||||
- [ ] 5.4 Figure-8 scenario
|
||||
- Implement `raft_sim_scenario_figure8`: a previous-term entry on a quorum, a leader change; assert the entry is NOT reported committed until a current-term entry commits (Leader Completeness / State Machine Safety preserved)
|
||||
- Observable completion: the previous-term entry's `commit_index` does not cover it until a current-term commit
|
||||
- _Requirements: 4.4_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 4.2_
|
||||
|
||||
- [ ] 5.5 Stale-rejoin scenario (bug-851 G5)
|
||||
- Implement `raft_sim_scenario_stale_rejoin`: a term-lagging node rejoins under a higher-term leader; assert it adopts the higher term from the AE/heartbeat path and does not re-elect at a reused term (Election Safety preserved)
|
||||
- Observable completion: the rejoining node's term advances via AE; no duplicate-term leader appears
|
||||
- _Requirements: 4.5_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 4.2_
|
||||
|
||||
## 6. CI batch and randomized seed sweep
|
||||
|
||||
- [ ] 6.1 RED+GREEN: randomized M-seed sweep with zero violations
|
||||
- Test: run `raft_sim_run` for a fixed `max_steps` across M seeds (M ≥ configured minimum) plus all targeted scenarios; fail if any seed/scenario violates any invariant
|
||||
- Implement the sweep harness in `test_raft.c`; bound `max_steps`, M, and array sizes so the target fits the library test-suite budget
|
||||
- Observable completion: the full batch runs clean and within budget
|
||||
- _Requirements: 6.1, 6.2, 6.4_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 5.1, 5.2, 5.3, 5.4, 5.5_
|
||||
|
||||
- [ ] 6.2 Verify the conformance batch is ASAN/UBSAN-clean
|
||||
- Run the conformance batch under the library test sanitizer build; fix any leak/overflow/UB in the harness (fixed-size arrays, no allocation)
|
||||
- Observable completion: `/xce-build lib` sanitizer run reports no memory errors, leaks, or UB for the conformance target
|
||||
- _Requirements: 6.3_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 6.1_
|
||||
|
||||
## 7. Deliberately-broken-build negative test (the oracle's teeth)
|
||||
|
||||
- [ ] 7.1 RED+GREEN: compile-time fault switch emulates the bug-851 defects in the adapter
|
||||
- Implement `RAFT_SIM_BREAK_TERM_STAMP` (adapter `on_persist_entry` writes term 0) and `RAFT_SIM_BREAK_TRUNCATE` (adapter `on_truncate_from` no-op), confined to the test/harness build configuration; default off
|
||||
- Observable completion: with a macro off, scenarios are clean; the macro is not set in the default CI run and does not touch `raft.c`
|
||||
- _Requirements: 7.1, 7.3_
|
||||
- _Boundary: raft_sim.c_
|
||||
- _Depends: 5.4, 5.5_
|
||||
|
||||
- [ ] 7.2 RED+GREEN: negative test asserts both directions
|
||||
- Test: under the broken configuration the Figure-8 / divergent-suffix scenario triggers a Log Matching / Leader Completeness / State Machine Safety violation (with trace dump); under the correct configuration the same scenario passes — both asserted in one test
|
||||
- Observable completion: the broken build is caught and the correct build passes, proving checker sensitivity
|
||||
- _Requirements: 7.1, 7.2, 7.4_
|
||||
- _Boundary: test_raft.c_
|
||||
- _Depends: 7.1_
|
||||
|
||||
## 8. Optional TLC-consumable trace export
|
||||
|
||||
- [ ]* 8.1 TLC-consumable trace export (deferred; off by default)
|
||||
- Implement `raft_sim_export_trace` gated by `RAFT_SIM_EXPORT`: per step, emit the action + tracked `raft.tla` variables (`messages` summary, per-node `currentTerm`/`state`/`log`/`commitIndex`) named per `raft/raft.tla`, in a stable line-oriented format; off by default
|
||||
- Observable completion: with `RAFT_SIM_EXPORT` set, a run emits a stable, spec-named record stream; default runs emit nothing extra
|
||||
- _Requirements: 8.1, 8.2, 8.3_
|
||||
- _Boundary: raft_sim.c, test_raft.c_
|
||||
- _Depends: 4.2_
|
||||
```
|
||||
|
|
@ -240,3 +240,64 @@ Mark tasks approved, record R5-2 (SHOULD) and the missing R6-1 test as known acc
|
|||
- **RN-1**: Confirm `XCE_SCHED_OP_UPDATE_RESVALS` (`scheduler.c:814`) is a live path (who sends `schedUpdateResvals` capnp field @8, and with which resources) before fixing the hardcoded slot 0 — it may be stub/dead code.
|
||||
- **RN-2**: Verify no out-of-tree or generated consumer (e.g., `src/build/lib/controller.capnp.c`) still references `ValueWithIdx` codecs before deleting the schema struct; a clean `ninja -C src/build` after removal settles this.
|
||||
- **RN-3**: EARS-R2-4 (resource add/remove mid-run) has no test harness today; determine whether the scheduler testsuite can drive `sched_handle_resources` followed by a worker push without a live controller.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
> POST-IMPLEMENTATION GAP AUDIT. Tasks 24/24 `[x]`, `spec.json.phase: implemented`.
|
||||
> Independent re-verification of R1–R6 against current `persistent_ha` code via codegraph + grep + verbatim reads. Supersedes the 2026-06-09 validation pass; records two status changes since then.
|
||||
|
||||
### Changes since the 2026-06-09 pass
|
||||
|
||||
1. **Process gap CLOSED.** `spec.json` now shows `phase: implemented` and `approvals.tasks.approved: true` (was `false`/`tasks-generated`). The prior "tasks unapproved" finding no longer applies.
|
||||
2. **Adjacent slot-0 defect FIXED.** `XCE_SCHED_OP_UPDATE_RESVALS` (per-host running update) at `src/scheduler/src/scheduler.c:816-834` now resolves `rv->name` via `key_find(sched->resmgr, ...)` and stores with `zarray_put(&w->resvals, r->idx, rv->val)` (line 831) — the hardcoded `zarray_put(..., 0, ...)` flagged on 2026-06-09 is gone. The last index-style hazard in the resval surface is closed. (Belongs to `scheduler-controller-msg-gap` scope, but worth recording: the defect class this spec targets is now eliminated across ALL resval paths A/B/C and both running-update handlers.)
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
| Req | EARS | Asset(s) — file:line | Tag |
|
||||
|-----|------|----------------------|-----|
|
||||
| R1 — name-keyed shared resvals in initial config | R1-1 | `src/schema/controller.capnp:241` `resvals @6: List(NameValue)`; `src/controller/dataex.c:381` `collect_resval(c->resmgr, &c->sharedinst, l)`; `src/controller/sched.c` cast to `xce_name_val_t**` | MET |
|
||||
| | R1-2 | `src/scheduler/src/scheduler.c:587-595` `sched_init_sharedInst`: `key_find(sched->resmgr, nv->name)` → `zarray_put(&sched->sharedInst, r->idx, val)` | MET |
|
||||
| | R1-3 | `scheduler.c:583-592`: NULL nv/name skipped; unknown name → `WARN` + `continue`, `sharedInst` untouched | MET |
|
||||
| | R1-4 | Same name→`r->idx` resolution as Path C (`scheduler.c:797-812`, `UPDATE_RESVALS_FULL`) | MET (by inspection; no equivalence test — see R6-1) |
|
||||
| R2 — name-keyed per-worker resvals to scheduler | R2-1 | `controller.capnp:140` `resvals @5: List(NameValue)`; `dataex.c:374` `collect_resvals_workers(c->resmgr, l)`; `sched.c:477` `collect_resval(c->resmgr, &w->host.resvals, ...)` | MET |
|
||||
| | R2-2 | `src/scheduler/src/schedobj.c:274-292` `make_sched_worker`: name lookup → `zarray_put(&worker->resvals, r->idx, val)` (line 291) | MET |
|
||||
| | R2-3 | `schedobj.c:279-288`: NULL skip; unknown name → `WARN` + skip; worker construction continues | MET |
|
||||
| | R2-4 | Resolution runs against live `sched->resmgr` on every receipt; no idx cached across messages (`schedobj.c:274-292`) | MET (by design; no mid-run add/remove test — RN-3) |
|
||||
| R3 — name-keyed worker resvals to API server | R3-1 | `src/controller/apiserver.c` `apiserver_worker_change`: same `collect_resval` + `xce_name_val_t**` path as scheduler push | MET |
|
||||
| | R3-2 | `src/frontend/apiserver/backend/convert.go:100-105` iterates `worker.resvals_to_sched` (`unsafe.Slice`), `convert.go:283` `cNameValToWorkerResval` (name-keyed, nil-checked); `convert_pure.go:210` `ResourceValues: r.resvals`; `models/hosts.go:94` `ResourceValues []ResourceDistributionInfo` | MET |
|
||||
| R4 — Path C unchanged | R4-1 | `src/scheduler/src/handler.c` (`Data_data_resvals` → `XCE_SCHED_OP_UPDATE_RESVALS_FULL`) and `scheduler.c:797-814` untouched, still `xce_resval_t` name-keyed | MET |
|
||||
| | R4-2 | `xce_conf4sched_t.resvals` (`typemap.h`) and `xce_worker_t.resvals_to_sched` (`types.h`) are `xce_name_val_t**`; codec auto-generated; `free_name_val_list` (`src/lib/utils.c:388`); all 7 controller free sites converted (codegraph: 7 callers across apiserver.c, dataex.c, opget.c, reply.c, sched.c) | MET |
|
||||
| R5 — remove index-keyed call sites | R5-1 | Zero cross-subsystem index-keyed transfers remain (grep: `collect_resval_with_idx`/`free_resval_with_idx_list` = 0 matches in src/ excl. build/); both receivers resolve by name | MET |
|
||||
| | R5-2 | `xce_val_with_idx_t` typedef (`src/include/xce/types.h:38-41`) and capnp `struct ValueWithIdx` (`controller.capnp:178`) still present — zero non-generated users, **no scoping comment** | Missing (SHOULD-level) |
|
||||
| R6 — no regression / correctness under divergent indices | R6-1 | Correct by code inspection (name resolution is order-independent at `scheduler.c:587` and `schedobj.c:283`); scheduler testsuite (`src/scheduler/testsuite/test.c`) has **zero** references to `sched_init_sharedInst` / `make_sched_worker` / divergent resval order | Missing (test only; code MET) |
|
||||
| | R6-2 | Same-order behavior preserved; controller/scheduler suites passed at task 8.2 | MET |
|
||||
| | R6-3 | `calc_wanted_*` read `sharedInst` / `worker->resvals` via scheduler-side `r->idx`; all writers (`sched_init_sharedInst`, `make_sched_worker`, both UPDATE_RESVALS handlers) now key by `r->idx` | MET |
|
||||
|
||||
**Score: 21/23 EARS MET. 2 Missing — both non-MUST (R5-2 SHOULD; R6-1 test-only, code present).** All MUST-level acceptance criteria are demonstrably satisfied.
|
||||
|
||||
### Residual Gaps
|
||||
|
||||
- **R5-2 (SHOULD):** Dead `xce_val_with_idx_t` / `struct ValueWithIdx` linger without the "removed or restricted to intra-subsystem use, with a comment explaining its scope" that EARS-R5-2 requests. Invites accidental reuse of the very index-keyed shape this feature eliminated.
|
||||
- **R6-1 (test gap):** The feature's entire raison d'être — correctness when controller and scheduler register resources in *different* orders — has no automated test. Code is order-independent by construction, but the invariant is unguarded against regression.
|
||||
- **Carry-forward note:** RN-3 (EARS-R2-4 mid-run resource add/remove) still has no harness. Lower priority than R6-1.
|
||||
|
||||
### Options for Open Work
|
||||
|
||||
**Option A — Close residuals in-spec (Recommended).**
|
||||
Delete `xce_val_with_idx_t` (types.h:38-41) + `struct ValueWithIdx` (controller.capnp:178) [or add the EARS-R5-2 scoping comment if retained for tooling], and add one table-driven scheduler testsuite case feeding `sched_init_sharedInst` / `make_sched_worker` a name-val list against a resmgr populated in reversed order, asserting `sharedInst` / `worker->resvals` slot contents.
|
||||
- Effort: **S** (~2 files + 1 test). Risk: **Low**.
|
||||
- Caveat: deleting from `types.h` is a shared-header edit — requires a clean `nix develop --command ninja -C src/build` to confirm no generated/out-of-tree consumer references `ValueWithIdx` (RN-2).
|
||||
|
||||
**Option B — Defer to a cleanup/test sweep across HA specs.**
|
||||
Bundle R5-2 + R6-1 + RN-3 with similar dead-type/test debt in adjacent specs.
|
||||
- Effort: **S-M**. Risk: **Low**. Cost: residue stays detached from this spec's history; may be dropped if the sweep stalls.
|
||||
|
||||
**Option C — Accept and close as-is.**
|
||||
Record R5-2 and the missing R6-1 test as known accepted debt; all MUST criteria met, slot-0 hazard already fixed, spec.json fully approved.
|
||||
- Effort: **none**. Risk: **Low-Medium** — dead type invites reuse; the feature's core scenario remains unverified by automation.
|
||||
|
||||
### Verdict
|
||||
|
||||
Feature is **functionally complete and CLOSED for all MUST requirements** (R1-1..R1-3, R2-1..R2-4, R3-1/R3-2, R4-1/R4-2, R5-1, R6-2/R6-3). Two non-blocking residuals remain (R5-2 dead-type cleanup; R6-1 divergent-order test). Recommend **Option A** to fully satisfy the SHOULD clause and lock the core invariant under test. Effort **S**, Risk **Low**.
|
||||
|
|
|
|||
|
|
@ -206,3 +206,99 @@ All 19 NotifyData event tags:
|
|||
- **R-2:** Confirm the notify-driven `schedUpdateJob` actually reaches `sched_update_existing_job()`: the dequeue `XCE_SCHED_OP_UPDATE_JOB` case calls `sched_update_jobs()` (bulk path); verify `sched_update_jobs` performs update-in-place for existing jobids (design Component 5 said it should) rather than duplicating jobs.
|
||||
- **R-3:** Is there a planned controller-side emitter for `Data_data_resources`/`Data_data_resvals` pushes to the scheduler (e.g., post-failover config resync in the persistent_ha work)? If not, decide whether Req 1's handlers stay forward-looking or get an emitter spec.
|
||||
- **R-4:** Whether `makeInstance(&resobj_type, r)` deep-copies all of `xce_resource_t` (name string included) — determines whether resource row payloads can be freed immediately after `makeResource` returns.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
> Post-implementation audit. tasks.md = 36/36 `[x]` (9.9 gated), phase=implemented. Method: every EARS AC re-verified against current on-disk source via codegraph (handler.c, scheduler.c, schedobj.c, plugin.c, helper.c) + grep. This pass tracks the 2026-06-09 G-1..G-7 findings: **G-1 and G-2 are now FIXED; G-3, G-4, G-5, G-6, G-7 remain OPEN; one NEW finding (G-8) supersedes the 2026-06-09 R-2 caveat.**
|
||||
|
||||
### 0. Delta vs 2026-06-09 audit
|
||||
|
||||
| Prior gap | 2026-06-18 status | Evidence |
|
||||
|---|---|---|
|
||||
| G-1 UPDATE_RESVALS clobbered worker index 0 | **FIXED** | scheduler.c:826-831 now resolves `rv->name`→`key_find(sched->resmgr)`→`r->idx`, `zarray_put(&w->resvals, r->idx, rv->val)`; unknown resource WARN+skip. Matches schedobj.c:283-291. |
|
||||
| G-2 UPDATE_RESOURCES could not update existing resource | **FIXED** | scheduler.c:775-792: on `makeResource()`==NULL (EEXIST), `key_find` the existing resource and copy `valtype/ordtype/flags/interval`; WARN on update and on lookup-miss. "Add-or-update" now satisfied for mutable scalar fields. |
|
||||
| G-3 killer/dequeue never free new-op payloads (leak) | **OPEN** | `sched_pend_update_killer` (scheduler.c:311-320) still has empty `break;` for UPDATE_QUEUE / UPDATE_RESOURCES / UPDATE_RESVALS_FULL / UPDATE_RESVALS / JOB_RUSAGE. Dequeue cases (scheduler.c:767-847) free nothing. Ownership IS transferred (handler.c:297-316 nullifies `notify->data.*`; sched_handle_dataset:130-145 nullifies rows) ⇒ confirmed per-message leak. |
|
||||
| G-4 job update copies only `state` | **OPEN (worse — see G-8)** | `sched_update_existing_job` (scheduler.c:707-725) still copies only `job->state`; priority/queue/resreq untouched. AND it is never reached (G-8). |
|
||||
| G-5 rusage not stored on job | **OPEN** | `xce_sched_job_t` (sched/types.h:80-96) has NO rusage field. JOB_RUSAGE case (scheduler.c:836-847) calls `plugin_job_rusage_update()` then drops+leaks the payload. UB-6.1 "store the usage data on the job object" unmet. |
|
||||
| G-6 test debt (9.2/9.3/9.4/9.6/9.7 `[x]` but absent; 9.5 stub; 9.8 TEST_PASS) | **OPEN** | testsuite/test.c still defines a LOCAL stub `sched_update_existing_queue` (test.c:25-34) — real schedobj.c impl never linked. Only real new tests: 9.1 notify-event-name (test.c:399-423) + enum compile check. No tests for any enqueue handler, dequeue case, `sched_update_existing_job`, `sched_op_name`, leaks. |
|
||||
| G-7 doc drift: `xce_sched_op_name` in lib/helper per design/tasks 1.1,1.3 | **OPEN** | Implemented as static `sched_op_name()` in scheduler.c:685-705, NOT `xce_sched_op_name()` in lib/helper.c. design.md Component 8 + tasks.md 1.1/1.3 (both `[x]`) still describe the wrong location/name. |
|
||||
|
||||
### 1. NEW finding
|
||||
|
||||
- **G-8 (correctness, HIGH — supersedes R-2): `sched_update_existing_job()` is dead code; notify-driven job updates take the bulk CREATE path.** `grep -rn sched_update_existing_job src/` returns ONLY its definition (scheduler.c:707) — zero callers. The `XCE_SCHED_OP_UPDATE_JOB` dequeue case (scheduler.c:755-758) routes to `sched_update_jobs()` (scheduler.c:629-642), which for every job calls `make_sched_job()` (a fresh `calloc`) + `sched_add_job()` (`key_add` into `sched->jobs`). For an existing jobid this either duplicate-keys or replaces the live `xce_sched_job_t` with a new object — discarding `allocs`/`accounts`/`schedobj` runtime state and leaking the old node. There is no `-ENOENT` for an unknown jobid (UB-3.2 unmet) and no in-place mutation (UB-3.1 unmet). Req 3 is effectively NOT closed at the dequeue layer despite the handler-side wiring and the `[x]` on tasks 5.6 / 3.1 / 3.2.
|
||||
|
||||
### 2. Requirement-to-Asset map (tagged)
|
||||
|
||||
| Req / AC | Asset (file:line) | Tag |
|
||||
|---|---|---|
|
||||
| 1 / UB-1.1 resources add-or-update | handler.c:59-80; scheduler.c:770-796 | CLOSED |
|
||||
| 1 / UB-1.2 resvals→sharedInst by idx | handler.c:82-103; scheduler.c:797-815 | CLOSED |
|
||||
| 1 / UB-1.3 skip NULL rows | scheduler.c:774, 802-804 | CLOSED |
|
||||
| 1 / UB-1.4 -ENOMEM, free partial | handler.c:68-71,91-94 | CLOSED |
|
||||
| 2 / UB-2.1, UB-2.2 dead pushmsg gone | handler.c:333-362 (no pushmsg case); grep: no `sched_handle_jobpush`/`workerpush` | CLOSED |
|
||||
| 3 / UB-3.1 update mutable fields | scheduler.c:707-725 (`state` only) AND unreachable | OPEN (G-4 + G-8) |
|
||||
| 3 / UB-3.2 -ENOENT unknown jobid | scheduler.c:716-718 — never reached; live path `sched_update_jobs` create-anyway | OPEN (G-8) |
|
||||
| 3 / UB-3.3 -EINVAL NULL payload | handler.c:181-183 | CLOSED (enqueue layer) |
|
||||
| 4 / UB-4.1/4.2/4.3 queue update | schedobj.c:485-511; scheduler.c:767-769 | CLOSED |
|
||||
| 5 / UB-5.1 per-worker resvals | scheduler.c:816-835 (idx fix) | CLOSED |
|
||||
| 5 / UB-5.2 skip unknown host | scheduler.c:824-825 | CLOSED |
|
||||
| 5 / UB-5.3 zero values→0 | handler.c:236-238 | CLOSED |
|
||||
| 6 / UB-6.1 store rusage on job | scheduler.c:836-847 (plugin only; not stored) | OPEN (G-5) |
|
||||
| 6 / UB-6.2 -ENOENT unknown jobid | scheduler.c:840-841 | CLOSED |
|
||||
| 6 / UB-6.3 -EINVAL NULL rusage | handler.c:258-260 | CLOSED |
|
||||
| 7 / UB-7.1 WARN op+errno | scheduler.c:855-858 | CLOSED |
|
||||
| 7 / UB-7.2 unknown notify→-EINVAL | handler.c:318-320 | CLOSED |
|
||||
| 7 / UB-7.3 unknown body_which→-EINVAL | handler.c:350-352 | CLOSED |
|
||||
| 8 / UB-8.1 op name in req log | handler.c:331 | CLOSED |
|
||||
| 8 / UB-8.2 dataset table name | handler.c:109 | CLOSED |
|
||||
| 8 / UB-8.3 notify event name | handler.c:279 | CLOSED |
|
||||
| 8 / UB-8.4 UNKNOWN(%d) fallback | helper.c (xce_notify_event_name) + scheduler.c:703 | CLOSED |
|
||||
| 8 / UB-8.5 symbolic op name in WARN | scheduler.c:685-705 `sched_op_name` (static, not lib helper) | CLOSED-with-deviation (G-7) |
|
||||
|
||||
**Score: 17 / 23 ACs fully closed.** OPEN: UB-3.1, UB-3.2 (G-8/G-4), UB-6.1 (G-5). Cross-cutting non-AC defects: G-3 (leak), G-6 (test debt), G-7 (doc drift). Of the 8 top-level requirements, **Req 2, 4, 5, 7, 8 are fully CLOSED; Req 1 CLOSED; Req 3 OPEN; Req 6 PARTIAL (delivery yes, storage no).**
|
||||
|
||||
### 3. Residual / deferred gaps (priority order)
|
||||
|
||||
1. **G-8 (HIGH, correctness)** — notify `schedUpdateJob` hits the create path; `sched_update_existing_job` is dead code. Corrupts/duplicates live job state; no -ENOENT. Req 3 unmet.
|
||||
2. **G-3 (MED, leak)** — five new ops never freed despite ownership transfer. Unbounded slow leak in a long-lived daemon; aggravated by G-8 (old job node also leaked on overwrite).
|
||||
3. **G-5 (MED, requirement shortfall)** — rusage never stored on job (no field exists). UB-6.1 unmet.
|
||||
4. **G-4 (LOW, requirement shortfall)** — even once reachable, job update applies only `state`; priority/queue/resreq dropped.
|
||||
5. **G-6 (MED, test debt / tracking integrity)** — tasks 9.2/9.3/9.4/9.6/9.7 `[x]` but no tests in test.c; 9.5 tests a local stub, not real schedobj.c; 9.8 is TEST_PASS+comment. Checkboxes overstate coverage; none of the dequeue correctness fixes (G-1/G-2) are tested.
|
||||
6. **G-7 (LOW, doc drift)** — design.md Component 8 + tasks 1.1/1.3 name `xce_sched_op_name` in lib/helper; reality is static `sched_op_name` in scheduler.c.
|
||||
|
||||
### 4. Integration risks
|
||||
|
||||
- Req 1 handlers remain forward-looking: no controller emitter pushes `Data_data_resources`/`Data_data_resvals` to the scheduler today (controller/sched.c pushes only schedjobs/schedworkers; opget.c serves them only as get-responses). G-1/G-2 fixes are correct but untriggered in production — regression-prone because untested AND unexercised. (Prior R-3 still open.)
|
||||
- G-8 + G-3 interact: overwriting a live job via `sched_add_job` leaks the prior node's `schedobj`/`allocs`/`accounts`. Surfaces under ASAN only when a notify update arrives for an already-loaded job — exactly the persistent_ha post-failover resync scenario.
|
||||
|
||||
### 5. Options for open work (G-8, G-3, G-5, G-4, G-6, G-7)
|
||||
|
||||
**Option A — Minimal correctness patch.**
|
||||
- G-8: in dequeue `XCE_SCHED_OP_UPDATE_JOB`, branch on `njobs==1 && sched_get_job(jobid)!=NULL` → `sched_update_existing_job()`; else keep `sched_update_jobs()`. OR route the notify-only op to a distinct `XCE_SCHED_OP_*` so the single-job update never enters the bulk path.
|
||||
- G-4: copy priority/queue/resreq (+resreq realloc) into `sched_update_existing_job`, mirroring `sched_update_existing_queue` (schedobj.c:498-506).
|
||||
- G-3: add per-op `free_*` in `sched_pend_update_killer` (reuse src/lib/request.c deep-free helpers — confirm exports per prior R-1 against ASAN).
|
||||
- G-7: edit design.md/tasks.md to record the `sched_op_name` location/name deviation.
|
||||
- Defer G-5, G-6.
|
||||
- Effort **S–M**, Risk **Medium** (ownership/free work must pass ASAN; double-free risk).
|
||||
|
||||
**Option B — Close everything incl. requirement + tests.**
|
||||
- All of A, plus G-5 (add `xce_job_rusages_t* rusages` to `xce_sched_job_t` + free in `kill_sched_job` + store in JOB_RUSAGE case) and G-6 (new `testsuite/test_msg_handlers.c` linking scheduler objects + capnp headers — the friction that derailed Task 9 originally — replacing the local-stub 9.5 and TEST_PASS 9.8 with real assertions; un-check 9.2-9.8 until real).
|
||||
- Effort **M–L (~5-7 days)**, Risk **Medium-High** (new test binary + meson wiring; struct lifetime changes).
|
||||
|
||||
**Option C — Hybrid (recommended).**
|
||||
- Phase 1: A's correctness/leak fixes (G-8, G-4, G-3) with a failing test first where linkable; G-7 doc sync.
|
||||
- Phase 2: G-6 dedicated test program for enqueue+dequeue paths incl. the G-1/G-2 regressions.
|
||||
- Phase 3 (needs decision): G-5 — add job-side rusage storage, OR formally amend UB-6.1 to "deliver to plugins only" (zero code). Do not leave requirement and code disagreeing.
|
||||
- Effort **M**, Risk **Medium**; staged so correctness lands first and tracking becomes honest before requirement closure.
|
||||
|
||||
### 6. Recommendation
|
||||
|
||||
Do NOT treat the spec as truly `implemented`. Five top-level requirements (2,4,5,7,8) and Req 1 are genuinely closed; **Req 3 is broken at the dequeue layer (G-8) and Req 6 stores nothing (G-5)**. Adopt Option C. Re-open/annotate tasks 5.6 (job update wiring), 9.2-9.8 (test debt), and 1.1/1.3 (doc drift). G-8 and G-3 are defects against the approved spec and should block a clean `implemented` sign-off.
|
||||
|
||||
### 7. Research still needed
|
||||
|
||||
- **R-1 (carry-over):** Confirm exported `free_*` deep-free helpers in src/lib/request.c for `xce_queue_t` / `xce_resval_list_t` / `xce_job_rusages_t` / `xce_resource_t*[]` and validate under ASAN before wiring the killer (G-3).
|
||||
- **R-3 (carry-over):** Planned controller emitter for `Data_data_resources`/`Data_data_resvals` to scheduler (persistent_ha resync)? Determines whether Req 1 stays forward-looking.
|
||||
- **R-4 (carry-over):** Does `makeResource`/`makeInstance` deep-copy `xce_resource_t` (name incl.)? Determines if resource rows can be freed right after enqueue (G-3 scope).
|
||||
|
|
|
|||
|
|
@ -287,3 +287,60 @@ Skip a dedicated build; rely on the next `/xce-build` from any active spec (e.g.
|
|||
## Research Needed
|
||||
|
||||
- None for this spec. Single carry-forward verification item: green `/xce-build controller` run on current HEAD (worker.c/join.c unmodified since 93ed282, but HA-branch changes to ha.c/raft.c in the working tree share the controller test binary — confirm no link/test interference).
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
**Mode:** POST-IMPLEMENTATION GAP AUDIT (spec.json phase=implemented; tasks 12/12 [x]).
|
||||
**Method:** EARS requirement → on-disk code verification via codegraph + targeted Read. Cites file:line at current HEAD.
|
||||
**Scope reminder (roadmap line 26):** this spec covers **attribute/endpoint update on rejoin only**. The job-state resync (running/finished jobs re-reported on rejoin) is the SEPARATE, not-yet-started spec **worker-rejoin-resync** (planned, no effort). That is explicitly NOT a gap of THIS spec — see Boundary note below.
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
| Req | Statement (abbrev) | Asset (file:line) | Status |
|
||||
|-----|--------------------|-------------------|--------|
|
||||
| 1.1 | `key_find` before create-or-update | `handle_worker_join` brief READ lock + `key_find` — join.c:91-95 | **CLOSED** |
|
||||
| 1.2 | Branch on lookup result (new vs exists) | `if (w==NULL) init_worker … else rejoin_worker` — join.c:101-113 | **CLOSED** |
|
||||
| 1.3 | Reject on non-ENOENT `key_find` error | `if (rc<0 && rc!=-ENOENT){ … idx=rc; }` — join.c:97-100 | **CLOSED** |
|
||||
| 2.1 | Create new entry via `makeWorker` | `init_worker` builds tmp `xce_worker_t`, calls `makeWorker(c->workermgr,&tmp)` — worker.c:339-354 | **CLOSED** |
|
||||
| 2.2 | On create fail: log, set status, no `on_worker_join` | NULL→`ERROR`+`return -errno`; caller sets `reply.status=idx`, `if(idx<0) return idx` before `on_worker_join` — worker.c:351-353, join.c:104-105/138-147 | **CLOSED** |
|
||||
| 2.3 | Assign new idx to `priv->u.idx` | `priv->u.idx = idx;` before `on_worker_join` — join.c:146 | **CLOSED** |
|
||||
| 3.1 | Update `hostattrs` on rejoin | `rejoin_worker` frees+`makeString` hosttype/hostmodel, copies numeric attrs — worker.c:376-386 | **CLOSED** |
|
||||
| 3.2 | Update `ep` on rejoin | `apply_join_state` sets `w->ep = ep` (worker.c:295), invoked by rejoin path — worker.c:388 | **CLOSED** |
|
||||
| 3.3 | Preserve admin status bits | `admin_status = w->host.status & XCE_WORKER_ADMIN_MASK` (worker.c:374) restored `w->host.status |= admin_status` (worker.c:390); mask = LOCKEDU\|CLOSED\|HIDDEN\|CLOSED_BY_ADMIN — worker.c:275-277 | **CLOSED** |
|
||||
| 4.1 | Clear `XCE_WORKER_UNAVAIL` | `w->host.status &= ~XCE_WORKER_UNAVAIL;` — worker.c:294 | **CLOSED** |
|
||||
| 4.2 | Apply hconfig overrides (type/model/mxj slots/dispatch window/thresholds/bool res) | `apply_join_state` worker.c:297-334: hosttype/model "!"-guarded override, `set_host_slots(mxj, fallback maxcpus)`, dispatch_window dup, thresholds dup, `mark_host_res` loop | **CLOSED** |
|
||||
| 4.3 | Update host-attr resource values (type,model,cpuf,maxcpus,maxmem,maxswap,maxtmp,ndisks,slots) | `add_hostattrs_resvals` (worker.c:223-250, 8 attrs) + `set_host_slots` (slots) — called worker.c:307,312 | **CLOSED** |
|
||||
| 4.4 | Reject if `find_matched_hostconfig`==NULL, no workermgr mutation | fail-fast before any lock: `hconfig==NULL → reply.status=-errno/-EINVAL; return` — join.c:77-84 | **CLOSED** |
|
||||
| 5.1 | BEFORE before mutation, AFTER after | `HOOK_BEFORE_EVENT` at join.c:87-88 (before key_find); `HOOK_AFTER_EVENT` at join.c:120-121 (after create/update, outside lock) | **CLOSED** |
|
||||
| 5.2 | `on_worker_join` after successful join (both paths) | `on_worker_join(c, ep)` — join.c:147 (reached for both init and rejoin when idx>=0) | **CLOSED** |
|
||||
| 5.3 | Critical section as short as current | No outer workermgr WRITE lock; only brief READ lock for key_find (join.c:91-95); mutation under per-worker WRITE lock in init/rejoin (worker.c:356-358, 372-392) | **CLOSED** |
|
||||
| 5.4 | Existing tests pass; `add_worker` removed | Controller `add_worker` (old join path) deleted from worker.c & worker.h:11-14 (only `init_worker`/`rejoin_worker` declared); new `test_worker_join.c` registered in controller_test.c:37,166. NOTE: residual unrelated `add_worker` symbols exist — `add_worker_to_xlist` (misc.c:11) and worker-daemon `add_worker` (xcecd_worker_utils.c:12); both are distinct functions, not the refactored one | **CLOSED (verify build)** |
|
||||
|
||||
**Score: 16/16 acceptance criteria CLOSED in code.** All 12 tasks have on-disk evidence. Design's planned architecture (explicit branch, no outer write lock, static `apply_join_state`, `init_worker`/`rejoin_worker`) matches the implementation exactly.
|
||||
|
||||
### Implementation deltas vs design (not gaps — improvements/additions)
|
||||
|
||||
1. **`validate_host` gate added** (join.c:65-72) — rejects unknown hosts with `-EPERM` before hostconfig lookup. Not in the original design; a correct hardening, no requirement conflict.
|
||||
2. **`persist_commit_barrier` added** (join.c:129-136) — the success reply is gated on the join WAL entry being majority-committed (ha-commit-barrier-rollout Req 1.1-1.4). This is the cross-spec commit-durability integration; barrier failure becomes the error reply and skips `on_worker_join`, preserving Req 2.2/5.2 semantics. Landed after this spec's design was written; consistent with it.
|
||||
3. **`apply_join_state` linkage** is `static` in production, non-`static` under `#ifdef XCE_TESTING` (worker.c:283-289) — enables the unit test to call it directly while keeping production encapsulation. Matches design intent.
|
||||
4. **`reference(w)`/`dereference(w)` discipline** around the found worker (join.c:93-94, 109) — refcount safety across the lock release between key_find and rejoin. Correct; not specified but required given the read-lock-then-release pattern.
|
||||
|
||||
### Residual / Deferred Gaps
|
||||
|
||||
**No open requirement gaps for this spec.** All criteria are met in code. Residual items are verification-only or out-of-boundary:
|
||||
|
||||
- **R1 (verification-only, carried from prior audit): green build/test on current HEAD.** worker.c/join.c are unmodified by HA-branch work, but join.c now depends on `persist_commit_barrier` and the controller test binary is shared with HA-branch changes (ha.c/raft.c) in the working tree. Action: run `/xce-build controller` and confirm `test_worker_join` (7 cases) plus `test_commit_barrier_reply` pass and no link interference. Effort: S (~1 build). Risk: Low.
|
||||
- **R2 (latent, MEDIUM — was flagged in prior audit body): `apply_join_state` resource-update failures are swallowed.** `update_resvals` returns 0 unconditionally (worker.c:133-146; logs via `PERROR` but discards rc), and `mark_host_res` rc in the `apply_join_state` loop (worker.c:333-334) is ignored. So Req 4.3 "update all resource values" can partially fail silently and the join still reports success. This MATCHES pre-refactor behavior (Error Handling table, design.md:423 — "log with PERROR; continue"), so it is **not a regression and not a spec gap** — it is a pre-existing design-accepted behavior. Logged here only as a known limitation.
|
||||
- **R3 (out-of-boundary — DO NOT count as this spec's gap): worker job-state resync on rejoin.** `rejoin_worker` updates attributes/endpoint/resources only. It does NOT recover running/finished job facts that were lost during the disconnect/failover window. `xce_join_req` carries only `hostattrs_t` (no job state). This is owned by the separate **worker-rejoin-resync** spec (roadmap line 26: "planned long ago, no effort yet"; dependencies: ha-consistency-invariants commit barrier + idempotency guards). Roadmap explicitly records this spec as "Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only)." **Boundary is clean and intentional.**
|
||||
|
||||
### Options for the one actionable open item (R1)
|
||||
|
||||
| Option | Description | Effort | Risk |
|
||||
|--------|-------------|--------|------|
|
||||
| **A (recommended)** | Run `/xce-build controller` on current HEAD; confirm `test_worker_join` + `test_commit_barrier_reply` green and no link interference with HA-branch ha.c/raft.c changes. Record result; close R1. | S (~1 build cycle) | Low |
|
||||
| **B** | Defer until next controller build for any reason; treat R1 as a standing CI gate rather than a spec action. | None now | Low (build drift risk if join.c's `persist_commit_barrier` dep changes upstream) |
|
||||
| **C** | Additionally harden R2: make `update_resvals`/`mark_host_res` propagate rc so a join with failed resource setup fails loud (per fail-loud preference). Re-scopes Req 4.3 error semantics — would need a spec amendment since current behavior is design-accepted. | M (touches worker.c shared helpers + tests; affects `restore_worker`/`mark_worker_unavail` callers) | Medium (behavior change to a shared path; out of original scope) |
|
||||
|
||||
**Recommendation: Option A.** The spec is functionally complete (16/16 criteria CLOSED); only a build/test confirmation on the integrated HEAD remains. R2 is a pre-existing accepted behavior — pursue Option C only if a separate fail-loud hardening spec is opened. R3 belongs to worker-rejoin-resync.
|
||||
|
|
|
|||
154
.kiro/specs/xce-test-e2e-primitives/design.md
Normal file
154
.kiro/specs/xce-test-e2e-primitives/design.md
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
# Design Document — xce-test-e2e-primitives
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose:** Make the `xce_test` client usable as a script-driven E2E driver: a piped command sequence whose **process exit code is the pass/fail verdict**, able to drive node lifecycle via a shell-escape, wait for failover deterministically, and assert outcomes in-tool.
|
||||
|
||||
**Users:** Test/CI authors and maintainers writing failover/durability/cold-restart E2E scenarios (e.g. the `test-commit-durability.sh` soak).
|
||||
|
||||
**Impact:** Today `xce_test` is an interactive client: `execute_line()` parses a line into argv, dispatches to a `command_t.func` (returning `int`, `<0` = error), and the main loop logs failures but **continues and exits 0**. Success is only inferable by grepping prose, and `jobsub` can print a garbage JobId (bug-786). This feature adds: session failure tracking + a real exit code; a `!` shell-escape; a fixed auto-variable substitution pass (`$?`, `$jobid`, `$jobcount`); `wait-primary`, `expect-jobs`, `expect-job`, `echo` commands; `#`/blank-line tolerance; and a `jobsub` reply-status fix. Everything lives in the single test-client TU; the shell remains the orchestrator.
|
||||
|
||||
### Goals
|
||||
- The process exit code reflects whether any command failed (Req 1).
|
||||
- A `!cmd` shell-escape lets one script drive node lifecycle; its status feeds `$?` (Req 2).
|
||||
- A fixed, tool-maintained auto-variable set (`$?`, `$jobid`, `$jobcount`) — no user variables/expressions (Req 3).
|
||||
- `wait-primary` blocks on the existing client failover until an op succeeds (Req 4).
|
||||
- `expect-jobs`/`expect-job` assertions fail the session on mismatch (Req 5).
|
||||
- `jobsub` reports success only on a clean ack with a valid JobId (Req 7).
|
||||
|
||||
### Non-Goals
|
||||
- Structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside `xce_test`; user-assignable variables, expressions, conditionals, or loops (the shell owns these).
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
- The `xce_test` command loop's **failure tracking + process exit code**, an **abort-vs-continue** mode flag, and `#`/blank-line handling.
|
||||
- A **line pre-processing step** in the client that (a) recognizes a leading `!` as a shell-escape and (b) substitutes the fixed auto-variables before tokenizing.
|
||||
- The fixed **auto-variable state** (`$?`, `$jobid`, `$jobcount`) and the points that set it.
|
||||
- New client commands: `wait-primary`, `expect-jobs`, `expect-job`, `echo`; the `set continue-on-error` toggle.
|
||||
- The **`cmd_jobsub` reply-status correctness** fix (bug-786).
|
||||
- The conversion of `src/tools/test-commit-durability.sh` to use these primitives.
|
||||
|
||||
### Out of Boundary
|
||||
- Controller/replication behavior and opcodes (consumed unchanged).
|
||||
- libxce client failover/master-rotation internals (consumed by `wait-primary`/`jobsub`/`joblist`).
|
||||
- Node lifecycle (deploy/kill/restart) — performed by the shell, reachable via `!cmd`.
|
||||
- Any general scripting language (variables/expressions/control flow).
|
||||
|
||||
### Allowed Dependencies
|
||||
- The existing `command_t`/`cmd_func_t` dispatch, `execute_line`, `parse_argv`, `find_command`, and the main read loop in `test.c`.
|
||||
- The existing client context (`active_ctx`), `cmd_jobsub`/`cmd_joblist` paths, and the libxce request/failover APIs they already call.
|
||||
- Standard C `system()` for the shell-escape; a monotonic clock helper for `wait-primary` timeout.
|
||||
|
||||
### Revalidation Triggers
|
||||
- Any change to the `cmd_func_t` contract (return-code semantics) → re-check all existing command handlers.
|
||||
- A change to how `jobsub`/`joblist` report acks or list jobs → re-check `$jobid`/`$jobcount` capture and the assertions.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
- `execute_line(char* line)`: `parse_argv` → `find_command(argv[0])` → `command->func(argc, argv)`; returns the handler's `int` rc (`<0` error).
|
||||
- Main loop: reads a line, `rc = execute_line(s)`; on `rc < 0` sets errno via `SET_ERROR` and **continues**; `main` returns 0. So failures never affect the exit code.
|
||||
- Commands return `int` (`0` ok, `<0` errno-style). `cmd_jobsub`/`cmd_joblist` already drive the client + libxce failover.
|
||||
|
||||
### Pattern
|
||||
Extend-in-place. Introduce a thin **pre-processing + verdict** wrapper around the existing dispatch; the new commands are ordinary `command_t` entries. No new module, no new files.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
Loop[main read loop] --> Pre[preprocess line]
|
||||
Pre -->|leading '#'/blank| Skip[ignore]
|
||||
Pre -->|leading '!'| Shell[run via system; set $?]
|
||||
Pre -->|else| Subst[substitute $? / $jobid / $jobcount]
|
||||
Subst --> Exec[execute_line -> find_command -> func]
|
||||
Exec --> Track[update session-failed + $?]
|
||||
Shell --> Track
|
||||
Track -->|abort mode + failed| Stop[stop loop]
|
||||
Track --> Loop
|
||||
Loop --> Verdict[main returns 1 if any failure else 0]
|
||||
```
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Modified Files
|
||||
- `src/controller/testsuite/test.c` — all client changes:
|
||||
- **Session/verdict state** (file-scope): `g_session_failed` (int), `g_continue_on_error` (bool), and auto-vars `g_last_status` (int), `g_last_jobid[XCE_MAXNAME_LEN]` (string), `g_last_jobcount` (int).
|
||||
- **`preprocess_and_run(char* line)`** (new): handle blank/`#`; leading `!` → `system(rest)`, set `g_last_status` from `WEXITSTATUS`; else substitute auto-vars then call `execute_line`. Update `g_session_failed`/`g_last_status` from the result; honor abort-vs-continue. Called from the main loop in place of the direct `execute_line`.
|
||||
- **`substitute_autovars(line) → expanded`** (new): replace whole-token `$?`, `$jobid`, `$jobcount` with current values; a reference with no current value yields a command failure (Req 3.6).
|
||||
- **New command handlers**: `cmd_wait_primary`, `cmd_expect_jobs`, `cmd_expect_job`, `cmd_echo`, registered in the command table(s).
|
||||
- **`cmd_set`**: add `continue-on-error` toggle.
|
||||
- **`cmd_jobsub`**: gate the "ok" report on a successful reply status + valid JobId; on success set `g_last_jobid`; otherwise print a typed error and return `<0` (bug-786).
|
||||
- **`cmd_joblist`**: set `g_last_jobcount`; expose a reusable internal count/contains helper for the assertions.
|
||||
- **`main`**: return `g_session_failed ? 1 : 0`.
|
||||
- `src/tools/test-commit-durability.sh` — convert to drive `xce_test` via piped scripts using `wait-primary`/`expect-jobs`/`$jobid` and the exit-code verdict instead of prose-grepping + sanitizer filtering.
|
||||
|
||||
### New Files
|
||||
- None.
|
||||
|
||||
> Note: the process exit code is only a trustworthy verdict if a clean (leak-free) exit is possible. Reuse the existing test run convention (`ASAN_OPTIONS=detect_leaks=0`) for the soak; leak cleanup of the client is out of scope here and tracked separately.
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Intent | Req Coverage |
|
||||
|-----------|--------|--------------|
|
||||
| Verdict & loop wrapper (`preprocess_and_run`, `g_session_failed`, exit code) | Turn command outcomes into a process verdict; abort/continue | 1.1–1.6, 6.1, 6.2 |
|
||||
| Shell-escape (`!cmd` via `system`) | Drive node lifecycle / setup from a script | 2.1–2.4 |
|
||||
| Auto-vars (`substitute_autovars`, `g_last_*`) | Reference last status / JobId / job count | 3.1–3.6 |
|
||||
| `cmd_wait_primary` | Block until a client op succeeds (failover-aware) | 4.1–4.4 |
|
||||
| `cmd_expect_jobs` / `cmd_expect_job` | Assert job count / membership; fail session on mismatch | 5.1–5.4 |
|
||||
| `cmd_echo` | Progress/section messages | 6.3 |
|
||||
| `cmd_jobsub` fix | Report success only on clean ack + valid JobId; set `$jobid` | 7.1–7.3, 3.2 |
|
||||
| `test-commit-durability.sh` driver | Reference E2E using the primitives | 8.1–8.3 |
|
||||
|
||||
### Verdict & loop wrapper
|
||||
- **State (file-scope):** `g_session_failed`, `g_continue_on_error`, `g_last_status`, `g_last_jobid[]`, `g_last_jobcount`.
|
||||
- **Contract:** every executed command updates `g_last_status` and, on failure, sets `g_session_failed`. In default mode the loop stops on first failure; under `continue-on-error` it keeps going. `main` returns non-zero iff `g_session_failed`.
|
||||
|
||||
### Shell-escape
|
||||
- A line whose first non-whitespace char is `!` runs `system(rest_of_line)`. `g_last_status = WEXITSTATUS(rc)` (or non-zero on `system` failure). A non-zero status counts as a command failure.
|
||||
|
||||
### Auto-variables
|
||||
- Whole-token substitution of `$?` (→ `g_last_status`), `$jobid` (→ `g_last_jobid`), `$jobcount` (→ `g_last_jobcount`) before tokenizing. Unset `$jobid`/`$jobcount` → command failure (no garbage). No other variable syntax is recognized.
|
||||
|
||||
### cmd_wait_primary
|
||||
- `wait-primary [timeout_ms]`: loop a cheap client op (e.g. the `joblist`/probe path through `active_ctx`, which already does master rotation) until success or the timeout (monotonic clock); return 0 on success, `<0` on timeout. Default timeout when arg omitted.
|
||||
|
||||
### cmd_expect_jobs / cmd_expect_job
|
||||
- `expect-jobs N`: obtain the current job count (shared helper with `cmd_joblist`); equal → 0, else print expected/actual and return `<0`.
|
||||
- `expect-job <jobid>`: list jobs; present → 0, else print the missing id and return `<0`.
|
||||
|
||||
### cmd_jobsub (bug-786 fix)
|
||||
- Inspect the reply: only when the controller returns a success status with a well-formed JobId does it print `jobsubmit ok: JobId: <id>` and set `g_last_jobid`. Otherwise print a typed error (redirect / no-master / not-ready / timeout) and return `<0`. Never print "ok" with an uninitialized/garbage id.
|
||||
|
||||
## Error Handling
|
||||
- **Command failure** → `g_last_status != 0`, `g_session_failed = 1`; abort (default) or continue. Non-zero process exit at end.
|
||||
- **Shell-escape failure** → treated as a command failure via `$?`.
|
||||
- **Unset auto-var reference** → command failure with a clear message (never substitutes garbage).
|
||||
- **`wait-primary` timeout** and **assertion mismatch** → command failure with expected/actual reported.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit / behavioral (driven through `xce_test` itself; no controller needed for most)
|
||||
- Exit code: a script with all-passing commands exits 0; a script with a failing `expect-jobs` exits non-zero (Req 1.1–1.3).
|
||||
- Abort vs continue: a failing command mid-script stops execution by default; under `set continue-on-error` later commands still run and the final exit is non-zero (Req 1.4, 1.5).
|
||||
- Shell-escape: `!true` → `$?`=0, success; `!false` → `$?`≠0, session fails (Req 2.1–2.4).
|
||||
- Auto-vars: `$?` reflects the prior command; an unset `$jobid` reference fails rather than substituting garbage (Req 3.1, 3.6).
|
||||
- Ergonomics: `#` comment and blank lines are ignored; `echo` succeeds (Req 6).
|
||||
- `jobsub` reporting: a non-acked submit returns failure and does not set `$jobid`; never prints a malformed id (Req 7) — verifiable against a live single node or by simulating a non-ack.
|
||||
|
||||
### Integration / E2E (live `/xce-deploy-ha` cluster)
|
||||
- Reference driver: converted `test-commit-durability.sh` runs a scenario — `wait-primary`, `jobsub` (capture `$jobid`), `expect-jobs`, `!kill` the primary, `wait-primary`, `expect-job $jobid` — and gates on the `xce_test` exit code (Req 4, 5, 8). Adding a `!`-driven cold-restart-all + `expect-jobs` step exercises the durability path (would catch a bug-778-class regression).
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Component | Test |
|
||||
|-------------|-----------|------|
|
||||
| 1.1–1.6 | Verdict & loop wrapper | exit-code + abort/continue tests |
|
||||
| 2.1–2.4 | Shell-escape | `!true`/`!false` tests |
|
||||
| 3.1–3.6 | Auto-vars | `$?`/`$jobid` substitution + unset-fails tests |
|
||||
| 4.1–4.4 | cmd_wait_primary | live failover wait |
|
||||
| 5.1–5.4 | cmd_expect_jobs / cmd_expect_job | assertion pass/fail tests |
|
||||
| 6.1–6.3 | loop wrapper + cmd_echo | comment/blank/echo tests |
|
||||
| 7.1–7.3 | cmd_jobsub fix | non-ack returns failure; no garbage id |
|
||||
| 8.1–8.3 | test-commit-durability.sh | converted soak runs, exit-code-gated |
|
||||
```
|
||||
94
.kiro/specs/xce-test-e2e-primitives/requirements.md
Normal file
94
.kiro/specs/xce-test-e2e-primitives/requirements.md
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
This feature adds a small set of scripting primitives to the `xce_test` client so that a complete end-to-end scenario — submit work, kill the primary, wait for failover, restart/cold-restart nodes, and assert that committed work survived — can be driven from a single piped or scripted command sequence whose **process exit code is the pass/fail verdict**. Today `xce_test` is an interactive client whose success can only be inferred by grepping human-readable output (and which can emit a malformed JobId), so E2E scripts must filter prose, suppress sanitizer noise, and poll controller logs for cluster state. These primitives let the shell remain the orchestrator (node lifecycle, loops) while `xce_test` reliably reports outcomes.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: command-failure tracking and a process exit code that reflects it; a `!`-prefixed shell-escape command; a fixed set of tool-maintained auto-variables (`$?`, `$jobid`, optional `$jobcount`); a `wait-primary` blocking command; `expect-jobs` / `expect-job` assertion commands; script ergonomics (`#` comments, blank lines, `echo`); correct `jobsub` success/failure reporting (bug-786); converting `src/tools/test-commit-durability.sh` to use these primitives.
|
||||
- **Out of scope**: structured/JSON output; a controller status/probe opcode; node-lifecycle commands inside `xce_test` (kill/restart/deploy stay in the shell, reachable via `!cmd`); user-assignable variables, expression evaluation, conditionals, or loops (the shell owns these); any change to controller/replication behavior.
|
||||
- **Adjacent expectations**: `wait-primary` and `jobsub`/`joblist` rely on the existing libxce client-side master-rotation/failover already built into `xce_test`; this feature consumes that behavior and does not change it.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Exit code reflects command outcomes
|
||||
|
||||
**Objective:** As an E2E script author, I want the `xce_test` process to exit non-zero when any command in the session failed, so that a shell can use the exit code alone as the pass/fail verdict.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a command in the session fails, the xce_test client shall record the session as failed.
|
||||
2. When the session ends and at least one command failed, the xce_test client shall exit with a non-zero status.
|
||||
3. When the session ends and every command succeeded, the xce_test client shall exit with status zero.
|
||||
4. While default (abort-on-failure) mode is active, when a command fails, the xce_test client shall stop executing subsequent commands and exit non-zero.
|
||||
5. Where `set continue-on-error` has been issued, when a command fails, the xce_test client shall continue executing subsequent commands and still exit non-zero at session end.
|
||||
6. The xce_test client shall send all human-readable log and diagnostic output in a way that does not corrupt the determination of the exit-code verdict.
|
||||
|
||||
### Requirement 2: Shell-escape command
|
||||
|
||||
**Objective:** As an E2E script author, I want to run an arbitrary shell command from within an `xce_test` script, so that a single script can drive node lifecycle (kill, restart, deploy) and environment setup without a separate orchestrator.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a script line begins with `!`, the xce_test client shall execute the remainder of the line as a shell command.
|
||||
2. When the shell command exits zero, the xce_test client shall treat the `!` command as successful.
|
||||
3. If the shell command exits non-zero, then the xce_test client shall treat the `!` command as failed (subject to Requirement 1's exit-code and abort/continue behavior).
|
||||
4. When a `!` command completes, the xce_test client shall set `$?` to the shell command's exit status.
|
||||
|
||||
### Requirement 3: Tool-maintained auto-variables
|
||||
|
||||
**Objective:** As an E2E script author, I want a fixed set of auto-variables that later commands can reference, so that I can assert on outcomes without parsing client output in the shell.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When any command completes, the xce_test client shall set `$?` to that command's status (zero on success, non-zero on failure).
|
||||
2. When a `jobsub` command receives a successful acknowledgment, the xce_test client shall set `$jobid` to the acknowledged JobId.
|
||||
3. When a subsequent command references `$jobid` (e.g. `expect-job $jobid`), the xce_test client shall substitute the most recently acknowledged JobId.
|
||||
4. Where the `$jobcount` auto-variable is provided, when a `joblist` command completes, the xce_test client shall set `$jobcount` to the number of jobs returned.
|
||||
5. The xce_test client shall not provide user-assignable variables, arithmetic, or expression evaluation beyond substituting this fixed set of auto-variables.
|
||||
6. If a command references an auto-variable that has no current value, then the xce_test client shall fail that command rather than substitute an undefined or garbage value.
|
||||
|
||||
### Requirement 4: Wait for an elected primary
|
||||
|
||||
**Objective:** As an E2E script author, I want a command that blocks until the cluster has a serving primary, so that scripts can proceed deterministically after a kill or restart without fixed sleeps or log polling.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When `wait-primary` is issued, the xce_test client shall repeatedly attempt a client operation against the configured masters until one succeeds.
|
||||
2. When a client operation succeeds within the timeout, the xce_test client shall complete `wait-primary` successfully.
|
||||
3. If no client operation succeeds within the timeout, then the xce_test client shall fail `wait-primary`.
|
||||
4. Where a timeout argument is supplied (`wait-primary <timeout_ms>`), the xce_test client shall use it as the maximum wait; where it is omitted, the client shall use a default timeout.
|
||||
|
||||
### Requirement 5: Outcome assertions
|
||||
|
||||
**Objective:** As an E2E script author, I want assertion commands that fail the session on mismatch, so that durability/availability expectations are checked in-tool rather than by parsing output.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When `expect-jobs N` is issued and the current job count equals `N`, the xce_test client shall treat the assertion as passed.
|
||||
2. If `expect-jobs N` is issued and the current job count does not equal `N`, then the xce_test client shall fail the assertion (non-zero status) and report the expected and actual counts.
|
||||
3. When `expect-job <jobid>` is issued and the given JobId is present in the job list, the xce_test client shall treat the assertion as passed.
|
||||
4. If `expect-job <jobid>` is issued and the given JobId is absent, then the xce_test client shall fail the assertion and report the missing JobId.
|
||||
|
||||
### Requirement 6: Script ergonomics
|
||||
|
||||
**Objective:** As an E2E script author, I want comments, blank lines, and progress messages, so that scenario scripts are readable and self-documenting.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a script line's first non-whitespace character is `#`, the xce_test client shall treat the line as a comment and ignore it.
|
||||
2. When a script line is empty or whitespace-only, the xce_test client shall ignore it without error.
|
||||
3. When `echo <message>` is issued, the xce_test client shall emit the message to output and treat the command as successful.
|
||||
|
||||
### Requirement 7: Correct job-submission reporting
|
||||
|
||||
**Objective:** As an E2E script author, I want `jobsub` to report success only when a job was genuinely accepted, so that scripts never chase a job the cluster never committed (bug-786).
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When `jobsub` receives a successful acknowledgment carrying a valid JobId, the xce_test client shall report success and set `$jobid` to that JobId.
|
||||
2. If `jobsub` does not receive a successful acknowledgment with a valid JobId, then the xce_test client shall report a typed failure (e.g. redirect, no-master, not-ready, or timeout) and set a non-zero status.
|
||||
3. The xce_test client shall never report `jobsub` success with a malformed or uninitialized JobId.
|
||||
|
||||
### Requirement 8: Reference E2E driver
|
||||
|
||||
**Objective:** As a maintainer, I want the existing durability soak to use these primitives, so that there is a working reference for script-driven E2E and the primitives are exercised.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The `test-commit-durability.sh` soak shall be converted to drive `xce_test` using the exit-code verdict and the new primitives instead of grepping prose and filtering sanitizer output.
|
||||
2. When the converted soak runs against a healthy cluster, it shall report pass/fail using `xce_test` exit codes and the assertion commands.
|
||||
3. The converted soak shall preserve its existing coverage (rolling primary kill + restart, zero acknowledged-job loss across failover).
|
||||
21
.kiro/specs/xce-test-e2e-primitives/research.md
Normal file
21
.kiro/specs/xce-test-e2e-primitives/research.md
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
# Research / Discovery — xce-test-e2e-primitives
|
||||
|
||||
## Discovery type
|
||||
LIGHT (extension to an existing single-file test client). No external research needed.
|
||||
|
||||
## Key findings (from src/controller/testsuite/test.c)
|
||||
- Dispatch: `execute_line(line)` → `parse_argv` → `find_command(argv[0])` → `command->func(argc, argv)`; handlers return `int` (`<0` = errno-style error). Command table is `command_t {name, func, doc}` arrays per context.
|
||||
- Main read loop reads a line, `rc = execute_line(s)`, and on `rc<0` only `SET_ERROR(-rc)` + continues; `main` returns 0 → **failures never affect the exit code** (the core gap for scriptable E2E).
|
||||
- `cmd_jobsub`/`cmd_joblist` already drive the client via `active_ctx` and libxce client-side master-rotation/failover — `wait-primary` and the assertions can reuse these paths.
|
||||
- bug-786: `cmd_jobsub` can print `jobsubmit ok: JobId: 63-15--1-...` (uninitialized id) on a non-clean ack; controller then logs `handle_jobsubmit fail to lock job <garbage>`. Root: success is reported without verifying the reply status/jobid.
|
||||
|
||||
## Design decisions
|
||||
- **Extend-in-place, single TU.** Add a `preprocess_and_run` wrapper (blank/`#`/`!`/auto-var-substitution) + file-scope verdict state; new commands as ordinary table entries; `main` returns `g_session_failed ? 1 : 0`. No new files, no new opcode, no JSON.
|
||||
- **Fixed auto-vars only** (`$?`, `$jobid`, `$jobcount`) — substitution, not a variable system; unset reference fails (no garbage).
|
||||
- **Shell-escape via `system()`**, status → `$?`; node lifecycle stays in the shell.
|
||||
- **`wait-primary`** loops an existing failover-aware client op until success/timeout (monotonic clock).
|
||||
- Exit-code trustworthiness depends on a clean exit; reuse `ASAN_OPTIONS=detect_leaks=0` for the soak (client leak cleanup is separate/out of scope).
|
||||
|
||||
## Risks
|
||||
- `system()` shell-escape = arbitrary execution — acceptable for a dev/test client only.
|
||||
- Auto-var substitution must be whole-token and conservative to avoid surprising expansions.
|
||||
|
|
@ -74,6 +74,8 @@ if (rc < 0) {
|
|||
}
|
||||
```
|
||||
|
||||
**Controller daemons MUST NEVER self-terminate on an operational condition.** A controller exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`_exit()`/`abort()`/`tlog_exit()` as a designed response to an operational failure (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" here means: log a loud operator-visible ERROR (with the actionable boundary), enter a permanent non-promotable/fenced state, **stay alive, and keep retrying** so the node self-heals when conditions recover — never terminate (a self-exited controller needs manual recovery and degrades quorum meanwhile). `assert()`/abort is reserved for genuine programming-invariant violations only. Authoritative statement: `ha-async-io.md` (process-lifecycle invariant) and `src/controller/CLAUDE.md` ("Process Lifecycle"); precedent: `ha-commit-durability`'s `persist_recover_loud_fence_if_unrecoverable` + `recovery_unrecoverable` sticky fence.
|
||||
|
||||
## Input Validation Pattern
|
||||
|
||||
Validate at the controller boundary using a `preCheck{Resource}Request()` helper before touching services:
|
||||
|
|
|
|||
83
.kiro/steering/persistence.md
Normal file
83
.kiro/steering/persistence.md
Normal file
|
|
@ -0,0 +1,83 @@
|
|||
# WAL & Persistence Durability
|
||||
|
||||
How the controller turns appended state into durable, recoverable, majority-backed state. This is the persistence-layer companion to `ha-async-io.md` (HA protocol / replication) and the `src/controller/CLAUDE.md` Protocol Verification Checklist (the authoritative safety gate). Owning code: `src/controller/persist.c`, `src/include/persist.h`, `src/controller/commit.c`. Landed by spec `ha-commit-durability` (2026-06-18, closes bug-661).
|
||||
|
||||
## Core Invariant: Acknowledged ⇒ Durable on a Majority
|
||||
|
||||
When the controller replies *ok* to a client, the operation's WAL entry is `fdatasync`'d on a fault-tolerant majority (the acking member + ⌊N/2⌋ others). The reply path is unchanged — it is gated on the commit barrier; the durability guarantee comes from strengthening what "committed" *means*, not from a new gate.
|
||||
|
||||
The load-bearing rule: **a member contributes to the commit majority for an LSN only once that LSN is durable locally.** The primary counts its own copy at `durable_lsn`, never at the appended tip.
|
||||
|
||||
## `durable_lsn` — The Prefix Watermark
|
||||
|
||||
`persist_t.durable_lsn` (`_Atomic uint64_t`) is the highest LSN whose bytes are `fdatasync`'d locally. It is a **prefix watermark**, never a per-entry set: `durable_lsn ≥ L` implies every entry `≤ L` is durable (because the WAL is a contiguous single-writer log). Invariants:
|
||||
|
||||
- `durable_lsn <= current_lsn - 1` (you cannot be durable past what you have appended).
|
||||
- Monotonic non-decreasing **except** clamp-on-truncate (`persist_truncate_wal_above` clamps `durable_lsn` down to the new tip).
|
||||
- Advances **only after a successful `fdatasync`** — fail-loud: on `fdatasync` error the watermark holds and an ERROR is logged. Never advance a watermark on a failed sync.
|
||||
|
||||
Read it via `persist_durable_lsn(p)`. The commit engine's self-cap (`majority_candidate_locked` in `commit.c`) uses exactly this value.
|
||||
|
||||
## Group-Commit Flusher (the `fsync_worker` pattern)
|
||||
|
||||
`fsync_worker` is **demand-driven, not a timer.** On append, `persist_wal_async` pwrites under `p->lock`, releases, then signals the flusher. The flusher does **one** `fdatasync` covering all pending appends, advances `durable_lsn`, then wakes commit evaluation. This amortizes one sync across the high-volume `WAL_MODE_RETAIN` fact stream (job start/finish/fail/rusage).
|
||||
|
||||
Two correctness rules when touching this path:
|
||||
|
||||
- **Lost-wakeup guard is level-triggered** on the appended-vs-synced gap (`durable_lsn < current_lsn - 1`), re-checked under `p->lock` before waiting and after each flush — never edge-triggered. An entry must never be considered durable before its `fdatasync`.
|
||||
- **Commit wake-up via callback, not a back-reference.** The flusher calls `on_durable_advance(ctx)` (registered with `persist_set_durable_cb`, ctx = the `commit_engine_t`) **only when `durable_lsn` advanced, and after releasing `p->lock`**. Lock order is strictly `p->lock → ce->lock` (cycle-free; persistence never includes commit). On shutdown, **join the flusher before** `commit_engine_fini` or the callback can fire into freed state.
|
||||
|
||||
`--wal-sync-ms` (default `DEFAULT_WAL_SYNC_MS = 10`) is the **maximum batching delay**, a latency/throughput tunable — it is *never* a durability gap. An acked op is durable regardless of its value (a large value just widens the batch window; the commit barrier still waits for `durable_lsn`).
|
||||
|
||||
## Standby Persist-Before-ACK
|
||||
|
||||
A standby `fdatasync`s a replicated entry before it sends its WalAck. `persist_wal_write_replicated` rc-gates the inline sync and returns `-EIO` on failure; the caller only ACKs on `rc == 0`. A standby must never ACK a non-durable entry — that is half of the majority guarantee.
|
||||
|
||||
## Recovery Integrity Verdict
|
||||
|
||||
`replay_wal` / `persist_recover` distinguish a clean log end from corruption and surface a `recover_verdict_t`:
|
||||
|
||||
- `RECOVER_CLEAN` — valid CRC chain to a clean entry boundary, OR an empty first-boot WAL, OR a pre-allocated-zero tail, OR a snapshot sentinel + clean replay.
|
||||
- `RECOVER_TRUNCATED` — replay stopped on structural corruption (CRC chain break, CRC mismatch, short header/payload read).
|
||||
|
||||
The verdict is **monotonic** (init CLEAN; only ever set TRUNCATED; a later clean segment cannot un-set it) and is **structural integrity only** — "behind the cluster" is a separate concern handled by the freshness/promotion fence. Recovery never silently discards committed entries. Consume via `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` (these are NOT `XCE_TESTING`-gated — the promotion gate reads them in production).
|
||||
|
||||
## Durable Floor & Durable-Backed Freshness
|
||||
|
||||
On recovery, the member publishes its **durable floor** = highest intact replayed LSN (the WAL tip) via `persist_durable_floor(p)`. It is **assigned, not max'd**, so a truncating restart lands at the lower intact tip and never over-advertises. This floor is a durable *last-log index*, **not** the commit position.
|
||||
|
||||
Election freshness is then durable-backed (formula lives in `ha.c`/`cluster.c`, consumed from here):
|
||||
|
||||
- A **candidate advertises bare `persist_durable_floor`** for the LSN axis — it may only claim a position it can durably back.
|
||||
- A **voter floor = `max(durable_floor, last_known_committed_lsn)`** — durable tip with the committed-high-water backstop retained (bug-339).
|
||||
|
||||
Result: a truncated/behind node cannot win an election or pass the promotion gate; the most-durable reachable node always can (no livelock).
|
||||
|
||||
## What Persistence Does NOT Persist
|
||||
|
||||
- **Never write `committed_lsn` to disk.** Raft keeps the commit index volatile; durability is about the **log**, not the commit index. The post-election safe-commit point is re-derived from voter replies (`ha_compute_safe_committed_lsn`), never from a stale ctrlhello.
|
||||
- **LSN 0 is a reserved sentinel** — never a real entry; the first append is LSN 1. The commit engine rejects `lsn == 0`.
|
||||
|
||||
## Fail-Loud, Never Self-Exit
|
||||
|
||||
When a member cannot reach a committed-consistent state (recovery-truncated/behind, no healthy peer), it enters a **loud permanent fence**: log a loud operator-visible ERROR (truncation boundary vs cluster HWM), set the sticky `recovery_unrecoverable` flag, hold non-promotable, **stay alive, and keep retrying** — it self-heals when a peer returns. It MUST NEVER `exit`/`abort` as a designed response. The decision is the pure `persist_recover_decision` → `RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}`; the action is `persist_recover_loud_fence_if_unrecoverable` (no terminal). Authoritative statement: `error-handling.md` + `src/controller/CLAUDE.md` "Process Lifecycle".
|
||||
|
||||
## Testing Durability Deterministically
|
||||
|
||||
- **Force durability synchronously** with `persist_flush(p)` (does an `fdatasync` and advances `durable_lsn`) when a test needs `durable_lsn ≥ L` before exercising a commit. The demand-driven flusher is async, so any test that synchronously expects "covering ACK ⇒ committed advances" must first make the entry durable (mirror this in in-memory fixtures by priming `durable_lsn` to the appended tip).
|
||||
- **`XCE_TESTING` seams** (e.g. `persist_test_set_force_fsync_fail`, `persist_test_durable_lsn_on_disk`) are compiled out of production — verify exclusion with `nm`.
|
||||
- **Gate test runs on the ANSI-stripped `:FAIL:` count** from the test binary's stdout, not on meson's `Fail:0` summary (meson aggregates and hides Unity sub-group failures).
|
||||
- Run final durability validation on a **real fsync-able filesystem** — some sandbox `/tmp` mounts mis-handle `fdatasync`.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Concern | Symbol / Site |
|
||||
|---------|---------------|
|
||||
| Durable watermark | `persist_durable_lsn(p)` / `persist_t.durable_lsn` |
|
||||
| Durable floor (WAL tip) | `persist_durable_floor(p)` |
|
||||
| Commit self-cap | `majority_candidate_locked` (commit.c) caps self at `durable_lsn` |
|
||||
| Group-commit wake | `persist_set_durable_cb` → `on_durable_advance(ctx)` |
|
||||
| Synchronous flush (tests) | `persist_flush(p)` |
|
||||
| Recovery verdict | `persist_recovery_verdict(p)` / `persist_recovery_truncated(p)` |
|
||||
| Loud fence (no exit) | `persist_recover_loud_fence_if_unrecoverable`, `recovery_unrecoverable` |
|
||||
| Flush cadence (max batch delay) | `--wal-sync-ms`, `DEFAULT_WAL_SYNC_MS = 10` |
|
||||
|
|
@ -8,6 +8,17 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin
|
|||
|
||||
- **ha-consistency-invariants** — UMBRELLA CONTRACT (requirements + design + research approved/complete). Implementation decomposed 2026-06-11 into the four sub-specs below; the umbrella generates NO tasks itself. Each sub-spec brief references the umbrella docs as authoritative.
|
||||
- ha-wal-replication — implemented except task 5.4 (live 3-node integration test; needs `/xce-deploy-ha` cluster); sub-spec 4's E2E extends it
|
||||
- **libxce-failover** — CLIENT-SIDE transparent controller failover (requirements generated 2026-06-16). Wires all libxce production request paths through the (already-built-but-orphaned) `xce_request_with_retry()` primitive + adds multi-master rotation; classifies on the real controller status taxonomy `E_XCE_REDIRECT` / `E_XCE_NOMASTER` / `E_XCE_NOT_READY` / `E_XCE_STANDBY` (`errnox.h:22-25`); bounded retry → distinct cluster-not-ready (no more opaque EINVAL); includes one small controller change so redirect replies carry the primary host/port (schema fields already exist, `controller.capnp:568-569`, just unpopulated in `join.c`). Verified E2E via `xce_test` (the original reproduction). Closes the integration gap left by `ha-config-bootstrap` Req 7 (tasks 10/10.1/10.2 shipped only the standalone primitive). See bug-608. Dependencies: ha-config-bootstrap (primitive + error codes), ha-commit-barrier-rollout (write idempotency makes write-retry safe).
|
||||
|
||||
## Raft TLA+ Conformance — raft-library enhancement series (NEW 2026-06-21)
|
||||
|
||||
**Goal:** evolve the election-only `src/lib/raft/raft.{c,h}` into a STRICT implementation of `raft/raft.tla` (Ongaro canonical) covering the log + commit half, keeping the pure callback-driven design so it stays independently verifiable via the deterministic simulator (`raft_sim.c`) against the TLA+ invariants (Election Safety, Log Matching, Leader Completeness, State Machine Safety). Motivation: **bug-851** — the log protocol is currently hand-rolled across `persist.c`/`repl.c`/`ha.c`/`ha_convergence.c` and violates the TLA+ log invariants. Gap analysis (six gaps G1–G6): `.kiro/specs/ha-term-safety/TLA-gap-analysis.md`. Implement in dependency order; each spec lands with simulator coverage. The library core stays pure/synchronous/deterministic — all I/O (async vtask, fan-out replication, the sync commit barrier, snapshot transport) lives in controller-side callback adapters. Preserve as policy (do NOT canonicalize away): PreVote, priority leadership, synchronous commit-barrier + persist-before-ACK durability, "controller never self-terminates".
|
||||
|
||||
- [x] raft-log-replication — *(specs generated + auto-approved 2026-06-21: 8 reqs, 24 sub-tasks, cross-spec review PASSED. Ready for /kiro-impl.)* Extend the library to OWN the replicated log. TLA+ actions: `ClientRequest`, `AppendEntries`, `HandleAppendEntriesRequest`, `UpdateTerm`-on-AppendEntries. Per-entry term in the library log model (closes **G1**); `AppendEntries` `logOk` = `prevLogTerm` match (**G2**); conflict branch deletes the divergent `(term,lsn)` suffix (**G3**); ANY higher-term RPC incl. AppendEntries/heartbeat advances `currentTerm` and steps down (**G5**). New pure callbacks: `on_persist_entry`, `on_get_entry_term(lsn)`, `on_send_append_entries(peer, prevLogIdx, prevLogTerm, entries, commitIdx)`, `on_truncate_from(lsn)`; controller adapters implement them over the existing WAL/repl transport. Simulator: drive AppendEntries across partitions/restarts, assert Log Matching. Dependencies: raft-library (election core). Closes bug-851 G1/G2/G3/G5.
|
||||
- [x] raft-commit-safety — *(specs generated + auto-approved 2026-06-21: 6 reqs, 18 sub-tasks, cross-spec review PASSED.)* `AdvanceCommitIndex` with the Figure-8 current-term rule (commit `Max(agreeIndexes)` only if `log[Max].term == currentTerm`) (**G4**); `RequestVote`/PreVote freshness becomes `(lastLogTerm, lastLogIndex)` instead of LSN-only (**G6**). New callback `on_commit_advanced(lsn)` feeds the existing synchronous commit barrier / durability (persist-before-ACK preserved as policy). Simulator: assert Leader Completeness. Dependencies: raft-log-replication. Closes bug-851 G4/G6.
|
||||
- [x] raft-sim-conformance — *(specs generated + auto-approved 2026-06-21: 8 reqs, 23 sub-tasks, cross-spec review PASSED.)* Extend `raft_sim.c` into a conformance/verification harness driving the library through `raft.tla`-action-equivalent transitions (randomized + targeted) and asserting the four Raft safety invariants; optionally trace-check against TLC output of `raft.tla`. The "use the verified spec to help" payoff: a regression oracle for the consensus core. Dependencies: raft-log-replication, raft-commit-safety.
|
||||
|
||||
> **ha-term-safety** (requirements + design generated 2026-06-21; `.kiro/specs/ha-term-safety/`) is the XCE-side INTEGRATION + REGRESSION umbrella for this family: it wires the controller WAL / commit-barrier into the new raft callbacks (thin adapter — protocol delegated to the library), owns the **config-change marker** (R7 — config changes as `(term,lsn)` log markers; bytes via the existing out-of-band config-push transport, hash-verified against the marker), and holds the live **bug-851 chaotic-failover regression**. Its protocol invariants R1–R6 are realized by the library specs above (not re-implemented in the controller). **Scope: R1–R7 only.** The former R8 (config-upgrade full-snapshot consistency, bug-857) was **descoped 2026-06-21 to the independent `config-upgrade-snapshot` spec** (below), because it requires entity-bearing snapshots that do not exist yet. Next: `/kiro-spec-tasks ha-term-safety`.
|
||||
|
||||
## Specs (dependency order)
|
||||
|
||||
|
|
@ -17,14 +28,20 @@ XCE is in the persistence + HA transformation (branch `persistent_ha`). Governin
|
|||
- [x] ha-commit-barrier-rollout — classification table, commit-first-then-reply sweep, RETAIN facts, config barrier, idempotency guards. Dependencies: ha-commit-engine. (specs done 2026-06-11; **IMPLEMENTED 2026-06-13**: 16/16 tasks, validate gate GO, 508 controller tests green under ASAN. Fixed bug-447 (key_find found-rc leak — queue/host control was silently unpersisted+unreplicated in production) and bug-396 (worker fact loss on quorum-loss gate rejection); task 5.2 needed one remediation round (wired the missing production config-ACK ingestion — would have bricked multi-member config upgrades — + kept the barrier off the iomgr thread). Logged not-fixed: bug-448 (job_ctrl ref leak), bug-451 (RUSAGE va_list OOB read). Barrier-before-state-removal precondition for ha-single-leader-role satisfied)
|
||||
- [x] ha-single-leader-role — promotion no-op + own-term gate, transitional-state (the former fourth role) removal. Dependencies: ha-commit-engine, ha-commit-barrier-rollout (HARD: barrier before state removal). (specs done 2026-06-11: 2 reqs / 7 components / 11 sub-tasks)
|
||||
- [x] ha-convergence-recovery — demotion truncate+self-restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum logging. Dependencies: ha-commit-engine (E2E additionally needs ha-single-leader-role). (specs done 2026-06-11: 9 reqs / 7 components / 13 sub-tasks)
|
||||
- [x] ha-commit-durability — closes bug-661 (acknowledgment did not imply durability; committed jobs lost across failover). Makes acknowledged ⇒ durable on a majority: the primary counts its own copy toward the commit majority only at its durable LSN (`majority_candidate_locked` caps self at `persist_durable_lsn`, not the appended tip); `fsync_worker` repurposed timer→demand-driven group-commit flusher (one `fdatasync` over pending appends, advance `durable_lsn`, wake commit-eval) — absorbs the former "WAL group-commit batching" backlog item (see below); recovery integrity verdict (clean vs corruption-truncated WAL) + durable-backed freshness floor (candidate advertises bare `persist_durable_floor`, voter floor = `max(durable_floor, learned_committed)`) + pre-promotion eligibility fence. Establishes the **controllers-never-self-terminate** invariant: a recovery-truncated/behind member that cannot catch up is a LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — never `exit`/`abort`), self-heals on peer return (see `ha-async-io.md` process-lifecycle invariant + `src/controller/CLAUDE.md`). Dependencies: ha-commit-engine (consumes commit-position semantics), ha-wal-replication (standby ACK/catch-up), ha-commit-barrier-rollout (reply gate, unchanged). (specs done 2026-06-18; **IMPLEMENTED 2026-06-18**: 17/17 tasks (1.1–5.3), validate-impl GO, controller suite 0 failures under ASAN; deterministic ack-before-fdatasync E2E RED→GREEN + rolling-SIGKILL soak zero acked-job loss. Verified by tests + Protocol Verification Checklist, NOT by raft.tla (which assumes a durable log). Fixed bug-760 (adjacent `test_fact_hook_append` raced the async flusher — test made durable-deterministic via `persist_flush`). raft.tla durability-model extension deferred unless explicitly opted in.)
|
||||
|
||||
## Future Specs / Backlog (dependency order)
|
||||
|
||||
- [ ] config-upgrade-snapshot — **descoped from `ha-term-safety` R8 on 2026-06-21 (requirements generated; `.kiro/specs/config-upgrade-snapshot/`).** Closes **bug-857**: a configuration upgrade can leave members with DIVERGENT materialized state because today it does config-push + restart + per-node WAL replay under the new config (`config_repl.c` `do_upgrade_config_with_bytes` ~1001-1163), with no snapshot pinning an identical post-upgrade state (replay is config-dependent — `lost_and_found` re-homes orphaned entities). Approach: an upgrade materializes a **full entity-bearing state snapshot at the config-change marker's `(term,lsn)` boundary**, embeds the new config, and installs it on every member via the `snapshot_transfer_s` transport — no per-node re-derivation; supersedes `ha-term-safety` R7's interim config-push byte delivery for upgrades. **HARD dependency: `entity-bearing-snapshots`** (today snapshots are a 16-byte LSN sentinel; the entity payload does not exist) AND `ha-term-safety` (the config-change marker it anchors on). Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state). Promote to design once `entity-bearing-snapshots` lands.
|
||||
- [ ] component-failover-adoption — **recorded 2026-06-16; depends on libxce-failover (HARD).** Make the C daemons and the Go apiserver actually exercise libxce's transparent failover. The mechanism reaches them for free (worker/scheduler/controller link `libxce_dep`; apiserver uses it via cgo, `src/frontend/apiserver/backend/clib.go`), but two per-component integration items remain: (1) each component must hand libxce the **full master list** instead of a single endpoint (today even the CLI uses only `conf->masters[0]`, `client.c:122`); (2) bounded retry makes the already-blocking `xce_request` block **longer** (seconds of backoff) — for the async C daemons (worker `controller_connector.c`, scheduler `handler.c`) this must be verified **liveness-safe**, i.e. it must not stall a thread that also drives heartbeats (the "never block the iomgr thread" rule). Scope: per-component master-list config + retry policy + liveness-safe blocking + per-component E2E failover for **scheduler, worker, apiserver**. (`xce_test` is NOT here — it is the in-spec verification client of libxce-failover.) Promote via `/kiro-spec-init component-failover-adoption` after libxce-failover lands.
|
||||
- [ ] worker-rejoin-resync — **planned long ago, no effort yet (recorded 2026-06-10).** When a worker rejoins after disconnect or controller failover, it shall re-report its running/finished job states so the controller recovers facts whose notifications were lost. Evidence of the gap (ha-consistency-invariants research deep-dive §A): `xce_join_req` carries only `hostattrs_t` (src/lib/request.c:1154-1181) — no job state; worker notify is fire-and-forget with zero redelivery (bug-396), so any job start/finish/fail sent during a disconnect or failover window is permanently lost today. Scope sketch: extend the join/hello exchange (or a post-join message) with a running+recent job-state report; controller reconciles against its job table through the commit barrier (entries go through majority commit like any state change); requires the `mark_job_*` idempotency guards (shared need with ha-consistency-invariants' pending-facts queue). Complements — does not replace — the primary-side pending-facts queue: the queue covers quorum-loss windows while connected; resync covers disconnect/failover windows. **Elevated 2026-06-11**: designated as the official backstop in ha-consistency-invariants Req 2.8 / boundary context — facts the controller received are retained-and-retried until committed; facts that never arrived (or died with the controller) are recovered only by this resync. Dependencies: ha-consistency-invariants (commit barrier + idempotency guards). Extends: worker-join-refactor (implemented — covers attribute/endpoint update on rejoin only).
|
||||
- [ ] admin-config-tool — drives `OP_upgradeConfig` (prerequisite for config-upgrade Phase 2). Encode ClusterConfig YAML/JSON → Cap'n Proto binary, SHA-256 over the bytes, connect to primary (SSL, same CA), send `OP_upgradeConfig`, display reply (version tag, acked count, errors), poll `OP_probe` after restart. Suggested CLI: `xce-config push --controller <host:port> --config cluster.yaml`. MUST validate that user configs don't use the reserved name `"lost_and_found"` for any queue/host (defence-in-depth: controller enforces at runtime via `load_primary_config()` FATAL, tool prevents at authoring time; migration: pre-existing clusters must rename before upgrade — see config-replication-design.md §6.5/§7.1). _(From FUTURE_WORK.md §1 + §8, 2026-05-12.)_
|
||||
- [ ] lightweight rejoin re-attach (failover optimization) — **recorded 2026-06-16; deferred (YAGNI).** Today, on controller failover a reconnecting daemon does a full re-join and apiserver/scheduler receive **full** state (e.g. the whole job list), not a delta; for a large cluster this is sizable but rare. Decision (libxce-failover design review): accept full-state rejoin — it is simple, correct, and still cheaper than a full-history replace with no snapshot (cf. IBM LSF). Future optimization: a session-token "re-attach" hello that lets the new primary skip re-pushing already-replicated state, or a delta-only state transfer keyed on the consumer's last-known committed LSN. Delta reconciliation is non-trivial (the primary must know what each consumer already holds). Promote only if failover-rejoin latency/bandwidth is **measured** as a problem at scale. Dependencies: libxce-failover (rejoin hook), component-failover-adoption (daemon rejoin content), worker-rejoin-resync (worker job state).
|
||||
- [ ] job-array-single-wal-record — **recorded 2026-06-16.** Enhance controller & scheduler so a job-array submission persists, replicates, and pushes to the scheduler as a **single WAL record** rather than one record per array element. Today a job array fans out to N per-job state changes, so a 10k-element array means 10k WAL appends + 10k replicated commits + 10k scheduler pushes — amplifying commit-engine load, replication bandwidth, and failover replay time linearly in array size. Scope sketch: (a) controller writes one WAL entry describing the array (base spec + index range/count + per-index overrides) committed atomically through the commit barrier; (b) replication carries the single record so standbys reconstruct the whole array from one committed LSN; (c) scheduler receives the array as one push and expands locally. Must preserve per-element identity/state addressing after expansion (status, cancel, completion still operate per task id). Dependencies: ha-commit-engine + ha-commit-barrier-rollout (single atomic commit), ha-consistency-invariants (replicated scheduling state). Spec advised before implementation; promote via `/kiro-spec-init "job-array-single-wal-record"` when ready.
|
||||
- [x] ~~WAL group-commit batching~~ — **recorded 2026-06-17 as deferred; ABSORBED INTO ha-commit-durability 2026-06-18 (design-review reversal).** The `/kiro-validate-design` review surfaced that the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible and not covered by `job-array-single-wal-record` — so a pure per-entry synchronous design would penalize the dominant write path. Decision (with user): **do group-commit now** — repurpose `fsync_worker` from a timer into a demand-driven flusher (batch one `fdatasync` across pending appends, advance `durable_lsn`, wake commit-eval) as part of ha-commit-durability rather than deferring. No longer a separate backlog item.
|
||||
- [ ] primary step-down-on-quorum-loss — optional CP follow-on noted in ha-consistency-invariants brief (writes already blocked on quorum loss; step-down is an availability/clarity optimization).
|
||||
- [ ] read consistency / leader leases — Raft §8 read-only optimization; status queries may currently observe stale or provisional state (documented deviation in ha-consistency-invariants).
|
||||
- [ ] entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs is already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion.
|
||||
- [ ] entity-bearing snapshots ("Task 11" format) — snapshot files are currently a 16-byte LSN sentinel (persist.c:435-437); real entity serialization unblocks snapshot-based state transfer and faster recovery. **Cheaper than it looks (verified 2026-06-11)**: the dual-zarray COW machinery it needs is already built and unit-tested but dormant — `xzarray_freeze`/`xzarray_thaw` in the vtable (zarray.c:695-757, O(1) freeze, thaw with `should_retain` GC filter, test_zarray_cow.c) with zero production callers (kvmgr.c:297 `vfreeze = NULL /* wired in P0-2 */`). Note: single-generation COW — the snapshot consumer owns freeze/thaw; do not add second concurrent consumers. Slot-level only (snapshots array structure, not object contents — serialize item state under per-object locks during the freeze window). Once landed, also strengthens ha-consistency-invariants §C eviction option (iii) and faster rebuild-on-demotion. **Downstream (recorded 2026-06-21): `config-upgrade-snapshot` HARD-depends on this** — it needs an entity-bearing snapshot to deliver an identical materialized post-upgrade state to every member (bug-857).
|
||||
|
||||
## Hardening / Direct-Implementation Items
|
||||
|
||||
|
|
@ -36,6 +53,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
|
|||
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
|
||||
- [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_
|
||||
- [x] steering sync — `ha-async-io.md` synchronized to the three-state model + own-term gate (transitional-state row/role removed; heartbeat re-arm PRIMARY-only; PRIMARY no-quorum fact allowlist noted). Done as part of ha-single-leader-role task 6.1.
|
||||
- [ ] xce_test WAL/snapshot inspection tooling — extend the offline inspector family in `xce_test` (sibling of the shipped `waldump`, `src/controller/testsuite/waldump.{c,h}` + `cmd_waldump`). Two remaining commands: **(a) `snapdump <snap_dir|snap_file>`** — decode the `XCESNAP\0` header (8-byte LSN; `persist.c:1266` `load_snapshot`) and report whether the snapshot carries job/queue entities or is an LSN-only stub. This directly settles Open Fork #3 of the failover job-loss investigation (`.kiro/specs/ha-commit-durability/INVESTIGATION-failover-job-loss.md`): if snapshots are entity-less, any empty-rejoin/WAL-prune below committed is unrecoverable BY DESIGN. **(b) `walcompare <dirA> <dirB>`** — diff two nodes' WALs by `lsn → crc/jobid` to flag divergence/prefix-gaps between members instantly. Reuse the `waldump.c` reader/decoder; same zero-alloc capnp jobid decode + `c_capnproto_dep`/`zlib_dep` wiring already added to `xce_test_deps`. Nonzero exit on entity-less snapshot / divergence so both double as E2E assertions. _(recorded 2026-06-20; `waldump` already done.)_
|
||||
- [ ] xce_test shell command support — add `!<command>` prefix to `xce_test` test scripts to execute arbitrary shell commands inline during test runs. Enables test scripts to drive environment setup/teardown (e.g., kill a node, write a file, run a helper) without calling out to a separate harness layer. Scope: parser change in the xce_test runner to detect `!`-prefixed lines, fork/exec the remainder as a shell command, fail the test on non-zero exit, capture stdout/stderr to the test log.
|
||||
- [ ] scheduler framework enhancement — strengthen the scheduler subsystem: (a) improve job scheduling policy extensibility (pluggable queue disciplines beyond FIFO), (b) add priority-aware dispatch, (c) improve resource-aware scheduling (CPU/memory/GPU slot tracking per host), (d) wire scheduler decisions through the commit barrier so scheduling state is replicated and survives failover. Dependencies: ha-consistency-invariants (commit barrier). Spec advised before implementation; promote via `/kiro-spec-init "scheduler-framework"` when ready.
|
||||
|
||||
|
|
|
|||
3
.mcp.json
Normal file
3
.mcp.json
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
{
|
||||
"mcpServers": {}
|
||||
}
|
||||
3
.opencode.json
Normal file
3
.opencode.json
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
{
|
||||
"mcpServers": {}
|
||||
}
|
||||
479
.wolf/anatomy.md
479
.wolf/anatomy.md
|
|
@ -1,7 +1,7 @@
|
|||
# anatomy.md
|
||||
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-06-17T04:38:41.384Z
|
||||
> Files: 523 tracked | Anatomy hits: 0 | Misses: 0
|
||||
> Auto-maintained by OpenWolf. Last scanned: 2026-06-21T13:07:04.771Z
|
||||
> Files: 507 tracked | Anatomy hits: 0 | Misses: 0
|
||||
|
||||
## ./
|
||||
|
||||
|
|
@ -9,6 +9,7 @@
|
|||
- `.gitignore` — Git ignore rules (~88 tok)
|
||||
- `.mcp.json` (~7 tok)
|
||||
- `.opencode.json` (~7 tok)
|
||||
- `.xce-durability-acked.vIjLlJ` (~19 tok)
|
||||
- `CLAUDE.md` — OpenWolf (~1400 tok)
|
||||
- `package-lock.json` — npm lock file (~1039 tok)
|
||||
- `package.json` — Node.js package manifest (~1 tok)
|
||||
|
|
@ -19,7 +20,7 @@
|
|||
|
||||
- `settings.json` (~497 tok)
|
||||
- `settings.json.bak` (~464 tok)
|
||||
- `settings.local.json` (~186 tok)
|
||||
- `settings.local.json` (~199 tok)
|
||||
- `ucai-formatter-cache.local.json` (~17 tok)
|
||||
|
||||
## .claude/rules/
|
||||
|
|
@ -200,8 +201,8 @@
|
|||
## .codegraph/
|
||||
|
||||
- `.gitignore` — Git ignore rules (~47 tok)
|
||||
- `codegraph.db-shm` (~8739 tok)
|
||||
- `daemon.log` (~9499 tok)
|
||||
- `codegraph.db-shm` (~8738 tok)
|
||||
- `daemon.log` (~15642 tok)
|
||||
- `daemon.pid` (~44 tok)
|
||||
|
||||
## .kiro/settings/rules/
|
||||
|
|
@ -252,11 +253,25 @@
|
|||
|
||||
- `ucai-formatter-cache.local.json` (~17 tok)
|
||||
|
||||
## .kiro/specs/config-upgrade-snapshot/
|
||||
|
||||
- `brief.md` — Brief: config-upgrade-snapshot (~923 tok)
|
||||
- `requirements.md` — Requirements Document (~1533 tok)
|
||||
- `spec.json` (~147 tok)
|
||||
|
||||
## .kiro/specs/controller-status/
|
||||
|
||||
- `design.md` — Design Document — controller-status (`ctrlstatus`) (~7961 tok)
|
||||
- `requirements.md` — Requirements Document (~3158 tok)
|
||||
- `research.md` — Gap Analysis — controller-status (ctrlstatus) (~4457 tok)
|
||||
- `spec.json` (~608 tok)
|
||||
- `tasks.md` — Implementation Plan (~4996 tok)
|
||||
|
||||
## .kiro/specs/election-iomgr-refactor/
|
||||
|
||||
- `design.md` — Design Document: election-iomgr-refactor (~6671 tok)
|
||||
- `requirements.md` — Requirements Document (~2695 tok)
|
||||
- `research.md` — Research & Design Decisions (~8796 tok)
|
||||
- `research.md` — Research & Design Decisions (~11773 tok)
|
||||
- `spec.json` (~129 tok)
|
||||
- `tasks.md` — Implementation Plan (~3693 tok)
|
||||
|
||||
|
|
@ -265,16 +280,35 @@
|
|||
- `brief.md` — Brief: ha-commit-barrier-rollout (~626 tok)
|
||||
- `design.md` — Design: ha-commit-barrier-rollout (~10372 tok)
|
||||
- `requirements.md` — Requirements Document (~3181 tok)
|
||||
- `research.md` — Research & Design Decisions: ha-commit-barrier-rollout (~3672 tok)
|
||||
- `research.md` — Research & Design Decisions: ha-commit-barrier-rollout (~6381 tok)
|
||||
- `spec.json` (~341 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-commit-barrier-rollout (~5658 tok)
|
||||
|
||||
## .kiro/specs/ha-commit-durability/
|
||||
|
||||
- `brief.md` — Brief: ha-commit-durability (~2855 tok)
|
||||
- `design.md` — Design Document — ha-commit-durability (~15280 tok)
|
||||
- `INVESTIGATION-failover-job-loss.md` — Investigation: committed jobs lost across controller failover (bug-795 family) (~5299 tok)
|
||||
- `requirements.md` — Requirements Document (~2587 tok)
|
||||
- `research.md` — Gap Analysis — ha-commit-durability (~7773 tok)
|
||||
- `spec.json` (~196 tok)
|
||||
- `tasks.md` — Implementation Plan (~10936 tok)
|
||||
|
||||
## .kiro/specs/ha-commit-durability/repro-851/
|
||||
|
||||
- `ctrl-1.log` (~6116 tok)
|
||||
- `ctrl-2.log` (~3742 tok)
|
||||
- `ctrl-3.log` (~8148 tok)
|
||||
- `waldump-ctrl-1.txt` (~1383 tok)
|
||||
- `waldump-ctrl-2.txt` (~1383 tok)
|
||||
- `waldump-ctrl-3.txt` (~1395 tok)
|
||||
|
||||
## .kiro/specs/ha-commit-engine/
|
||||
|
||||
- `brief.md` — Brief: ha-commit-engine (~588 tok)
|
||||
- `design.md` — Design: ha-commit-engine (~7600 tok)
|
||||
- `requirements.md` — Requirements Document (~2715 tok)
|
||||
- `research.md` — Research Log: ha-commit-engine (~1345 tok)
|
||||
- `research.md` — Research Log: ha-commit-engine (~3398 tok)
|
||||
- `spec.json` (~255 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-commit-engine (~4593 tok)
|
||||
|
||||
|
|
@ -285,7 +319,7 @@
|
|||
- `election-trace.md` — Election Trace: 3-Node Cold Start (2026-06-04 00:15:58) (~2907 tok)
|
||||
- `manual-tests.md` — Manual Integration Tests: ha-config-bootstrap (~3748 tok)
|
||||
- `requirements.md` — Requirements: HA Config Bootstrap & Raft-Style Quorum Election (~3890 tok)
|
||||
- `research.md` — Cross-Spec Gap Audit (2026-06-03) (~14349 tok)
|
||||
- `research.md` — Cross-Spec Gap Audit (2026-06-03) (~16849 tok)
|
||||
- `spec.json` (~293 tok)
|
||||
- `tasks.md` — Implementation Plan: HA Config Bootstrap & Raft-Style Quorum Election (~5780 tok)
|
||||
|
||||
|
|
@ -294,7 +328,7 @@
|
|||
- `brief.md` — Brief: ha-consistency-invariants (~2916 tok)
|
||||
- `design.md` — Design: ha-consistency-invariants (~8316 tok)
|
||||
- `requirements.md` — Requirements Document (~3454 tok)
|
||||
- `research.md` — Gap Analysis: ha-consistency-invariants (~7683 tok)
|
||||
- `research.md` — Gap Analysis: ha-consistency-invariants (~10252 tok)
|
||||
- `spec.json` (~259 tok)
|
||||
|
||||
## .kiro/specs/ha-convergence-recovery/
|
||||
|
|
@ -302,21 +336,35 @@
|
|||
- `brief.md` — Brief: ha-convergence-recovery (~746 tok)
|
||||
- `design.md` — Design: ha-convergence-recovery (~9928 tok)
|
||||
- `requirements.md` — Requirements Document (~3586 tok)
|
||||
- `research.md` — Research & Design Decisions: ha-convergence-recovery (~3542 tok)
|
||||
- `research.md` — Research & Design Decisions: ha-convergence-recovery (~6912 tok)
|
||||
- `spec.json` (~142 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-convergence-recovery (~5679 tok)
|
||||
|
||||
## .kiro/specs/ha-elect-ctx-lifecycle/
|
||||
|
||||
- `brief.md` — Discovery Brief: ha-elect-ctx-lifecycle (~774 tok)
|
||||
- `research.md` — Research / Gap Analysis — ha-elect-ctx-lifecycle (~1092 tok)
|
||||
- `spec.json` (~130 tok)
|
||||
|
||||
## .kiro/specs/ha-elect-ctx-lifecycle/.claude/
|
||||
|
||||
- `ucai-formatter-cache.local.json` (~17 tok)
|
||||
|
||||
## .kiro/specs/ha-heartbeat-committed-propagation/
|
||||
|
||||
- `brief.md` — Brief: ha-heartbeat-committed-propagation (~1403 tok)
|
||||
- `design.md` — Design Document — ha-heartbeat-committed-propagation (~4689 tok)
|
||||
- `requirements.md` — Requirements Document (~1763 tok)
|
||||
- `research.md` — Research / Gap Analysis — ha-heartbeat-committed-propagation (~2474 tok)
|
||||
- `spec.json` (~301 tok)
|
||||
- `tasks.md` — Implementation Plan (~2708 tok)
|
||||
|
||||
## .kiro/specs/ha-heartbeat-liveness/
|
||||
|
||||
- `design.md` — Design Document: ha-heartbeat-liveness (~5409 tok)
|
||||
- `manual-tests.md` — Manual Integration Tests: ha-heartbeat-liveness — Task 4.5 (~1726 tok)
|
||||
- `requirements.md` — Requirements Document (~2054 tok)
|
||||
- `research.md` — Gap Analysis: ha-heartbeat-liveness (~5437 tok)
|
||||
- `research.md` — Gap Analysis: ha-heartbeat-liveness (~7901 tok)
|
||||
- `spec.json` (~171 tok)
|
||||
- `tasks.md` — Implementation Plan (~2227 tok)
|
||||
|
||||
|
|
@ -324,7 +372,7 @@
|
|||
|
||||
- `design.md` — Design Document: ha-peer-connection-reuse (~5369 tok)
|
||||
- `requirements.md` — Requirements Document (~1605 tok)
|
||||
- `research.md` — Research & Design Decisions: ha-peer-connection-reuse (~6598 tok)
|
||||
- `research.md` — Research & Design Decisions: ha-peer-connection-reuse (~8740 tok)
|
||||
- `spec.json` (~169 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-peer-connection-reuse (~2022 tok)
|
||||
|
||||
|
|
@ -332,7 +380,7 @@
|
|||
|
||||
- `design.md` — Design Document (~6905 tok)
|
||||
- `requirements.md` — Requirements Document (~3604 tok)
|
||||
- `research.md` — Gap Analysis: ha-prevote (~6077 tok)
|
||||
- `research.md` — Gap Analysis: ha-prevote (~8259 tok)
|
||||
- `spec.json` (~126 tok)
|
||||
- `tasks.md` — Implementation Plan (~3197 tok)
|
||||
|
||||
|
|
@ -341,7 +389,7 @@
|
|||
- `design.md` — Design: HA Replication Hardening (~5577 tok)
|
||||
- `raft-tla-audit.md` — Raft TLA+ Specification Audit (~1591 tok)
|
||||
- `requirements.md` — Requirements Document (~2850 tok)
|
||||
- `research.md` — Gap Analysis: HA Replication Hardening (~6234 tok)
|
||||
- `research.md` — Gap Analysis: HA Replication Hardening (~9401 tok)
|
||||
- `spec.json` (~198 tok)
|
||||
- `tasks.md` — Implementation Plan: HA Replication Hardening (~4501 tok)
|
||||
|
||||
|
|
@ -350,15 +398,24 @@
|
|||
- `brief.md` — Brief: ha-single-leader-role (~624 tok)
|
||||
- `design.md` — Design: ha-single-leader-role (~8611 tok)
|
||||
- `requirements.md` — Requirements Document (~2177 tok)
|
||||
- `research.md` — Research & Discovery Log: ha-single-leader-role (~3970 tok)
|
||||
- `research.md` — Research & Discovery Log: ha-single-leader-role (~5686 tok)
|
||||
- `spec.json` (~142 tok)
|
||||
- `tasks.md` — Implementation Plan: ha-single-leader-role (~5040 tok)
|
||||
|
||||
## .kiro/specs/ha-term-safety/
|
||||
|
||||
- `design.md` — Design Document (~11209 tok)
|
||||
- `requirements.md` — Requirements Document (~4240 tok)
|
||||
- `research.md` — Research & Design Decisions (~5375 tok)
|
||||
- `spec.json` (~128 tok)
|
||||
- `TLA-gap-analysis.md` — Gap Analysis: XCE HA implementation vs `raft/raft.tla` (~2070 tok)
|
||||
|
||||
## .kiro/specs/ha-wal-replication/
|
||||
|
||||
- `brief.md` — Brief: ha-wal-replication (~829 tok)
|
||||
- `design.md` — Design: HA WAL Replication (~1413 tok)
|
||||
- `requirements.md` — Requirements Document (~1087 tok)
|
||||
- `research.md` — Research: HA WAL Replication (~2542 tok)
|
||||
- `spec.json` (~128 tok)
|
||||
- `tasks.md` — Implementation Plan: HA WAL Replication (~1737 tok)
|
||||
|
||||
|
|
@ -367,50 +424,84 @@
|
|||
- `brief.md` — Brief: libxce-failover (~2238 tok)
|
||||
- `design.md` — Design Document — libxce-failover (~8279 tok)
|
||||
- `requirements.md` — Requirements Document (~2933 tok)
|
||||
- `research.md` — Research Log — libxce-failover (~2344 tok)
|
||||
- `research.md` — Research Log — libxce-failover (~4638 tok)
|
||||
- `spec.json` (~128 tok)
|
||||
- `tasks.md` — Implementation Plan (~3305 tok)
|
||||
- `tasks.md` — Implementation Plan (~3791 tok)
|
||||
|
||||
## .kiro/specs/raft-commit-safety/
|
||||
|
||||
- `brief.md` — Brief: raft-commit-safety (~967 tok)
|
||||
- `design.md` — Design Document (~8677 tok)
|
||||
- `requirements.md` — Requirements Document (~4523 tok)
|
||||
- `spec.json` (~450 tok)
|
||||
- `tasks.md` — Implementation Plan (~3186 tok)
|
||||
|
||||
## .kiro/specs/raft-library/
|
||||
|
||||
- `design.md` — Design Document: raft-library (~7756 tok)
|
||||
- `requirements.md` — Requirements Document (~2115 tok)
|
||||
- `research.md` — Research & Design Decisions: raft-library (~3768 tok)
|
||||
- `research.md` — Research & Design Decisions: raft-library (~6052 tok)
|
||||
- `spec.json` (~215 tok)
|
||||
- `tasks.md` — Implementation Plan (~4312 tok)
|
||||
|
||||
## .kiro/specs/raft-log-replication/
|
||||
|
||||
- `brief.md` — Brief: raft-log-replication (~1297 tok)
|
||||
- `design.md` — Design Document (~7854 tok)
|
||||
- `requirements.md` — Requirements Document (~4710 tok)
|
||||
- `research.md` — Research Log: raft-log-replication (~1206 tok)
|
||||
- `spec.json` (~307 tok)
|
||||
- `tasks.md` — Implementation Plan (~3226 tok)
|
||||
|
||||
## .kiro/specs/raft-sim-conformance/
|
||||
|
||||
- `brief.md` — Brief: raft-sim-conformance (~697 tok)
|
||||
- `design.md` — Design Document (~8051 tok)
|
||||
- `requirements.md` — Requirements Document (~3776 tok)
|
||||
- `spec.json` — Declares divergence (~428 tok)
|
||||
- `tasks.md` — Implementation Plan (~3337 tok)
|
||||
|
||||
## .kiro/specs/resval-id-decoupling/
|
||||
|
||||
- `design.md` — Design: resval-id-decoupling (~3914 tok)
|
||||
- `requirements.md` — Requirements: resval-id-decoupling (~2360 tok)
|
||||
- `research.md` — Gap Analysis: resval-id-decoupling (~5568 tok)
|
||||
- `research.md` — Gap Analysis: resval-id-decoupling (~7389 tok)
|
||||
- `spec.json` — Declares is (~226 tok)
|
||||
- `tasks.md` — Implementation Tasks: resval-id-decoupling (~2424 tok)
|
||||
|
||||
## .kiro/specs/scheduler-controller-msg-gap/
|
||||
|
||||
- `design.md` — Design: scheduler-controller-msg-gap (~4756 tok)
|
||||
- `design.md` — Design: scheduler-controller-msg-gap (~5006 tok)
|
||||
- `requirements.md` — Requirements: scheduler-controller-msg-gap (~2712 tok)
|
||||
- `research.md` — Research Log: scheduler-controller-msg-gap (~5630 tok)
|
||||
- `research.md` — Research Log: scheduler-controller-msg-gap (~8281 tok)
|
||||
- `spec.json` (~230 tok)
|
||||
- `tasks.md` — Tasks: scheduler-controller-msg-gap (~2334 tok)
|
||||
- `tasks.md` — Tasks: scheduler-controller-msg-gap (~4650 tok)
|
||||
|
||||
## .kiro/specs/worker-join-refactor/
|
||||
|
||||
- `design.md` — Design Document: worker-join-refactor (~4709 tok)
|
||||
- `requirements.md` — Requirements Document (~1401 tok)
|
||||
- `research.md` — Research Log: worker-join-refactor (~5421 tok)
|
||||
- `research.md` — Research Log: worker-join-refactor (~7581 tok)
|
||||
- `spec.json` (~128 tok)
|
||||
- `tasks.md` — Implementation Plan (~1638 tok)
|
||||
|
||||
## .kiro/specs/xce-test-e2e-primitives/
|
||||
|
||||
- `design.md` — Design Document — xce-test-e2e-primitives (~2913 tok)
|
||||
- `requirements.md` — Requirements Document (~2163 tok)
|
||||
- `research.md` — Research / Discovery — xce-test-e2e-primitives (~499 tok)
|
||||
- `spec.json` (~271 tok)
|
||||
- `tasks.md` — Implementation Plan (~4188 tok)
|
||||
|
||||
## .kiro/steering/
|
||||
|
||||
- `api-standards.md` — API Standards (~1291 tok)
|
||||
- `build-deployment.md` — Build & Deployment (~1241 tok)
|
||||
- `error-handling.md` — Error Handling Standards (~782 tok)
|
||||
- `ha-async-io.md` — HA Protocol & Async I/O (~1431 tok)
|
||||
- `error-handling.md` — Error Handling Standards (~1026 tok)
|
||||
- `ha-async-io.md` — HA Protocol & Async I/O (~1603 tok)
|
||||
- `persistence.md` — WAL & Persistence Durability (~2042 tok)
|
||||
- `product.md` — Product Overview (~1246 tok)
|
||||
- `roadmap.md` — Roadmap (~3412 tok)
|
||||
- `roadmap.md` — Roadmap (~6012 tok)
|
||||
- `security.md` — Security Standards (~804 tok)
|
||||
- `structure.md` — Project Structure (~1170 tok)
|
||||
- `tech.md` — Technology Stack (~1155 tok)
|
||||
|
|
@ -429,330 +520,238 @@
|
|||
|
||||
## graphify-out/
|
||||
|
||||
- `.graphify_labels.json` (~5517 tok)
|
||||
- `.graphify_labels.json` (~8807 tok)
|
||||
- `.graphify_root` (~1 tok)
|
||||
- `GRAPH_REPORT.md` — Graph Report - persistent_ha (2026-06-17) (~45196 tok)
|
||||
- `manifest.json` (~36616 tok)
|
||||
- `GRAPH_REPORT.md` — Graph Report - persistent_ha (2026-06-20) (~70951 tok)
|
||||
- `manifest.json` (~41636 tok)
|
||||
|
||||
## graphify-out/cache/
|
||||
|
||||
- `stat-index.json` (~46526 tok)
|
||||
- `stat-index.json` (~52786 tok)
|
||||
|
||||
## graphify-out/cache/ast/
|
||||
|
||||
- `002dac674489624d72176d5d0b18f4979ccec9aeb986aef479a18ce84ad824f7.json` (~269 tok)
|
||||
- `004cc8d07f2f670da29d7c7f144c8875f274b8438fb1d4306fb6cc676c745fea.json` (~1050 tok)
|
||||
- `005d423b344fb7001a76e9a01e7226b28957f0318f503b6acd0f7f72a3fcde8e.json` (~227 tok)
|
||||
- `0078fef375f65aef38dd21e3dcdc8c9930568572612ef8380cbe3fdfa1f8c710.json` (~9104 tok)
|
||||
- `008610a0f4315b9a68f7cde989a73b27de885c7ab2c96a9e7440be16cf2fa094.json` (~2422 tok)
|
||||
- `008e1a68f9305eaa5f4ea97dcbf0cadc3905e36d892968c53334f2b1f0fffcb6.json` (~30245 tok)
|
||||
- `00a5ff6db9d63a87d19ca8b0a68d7105c6ea2fa89be5f94c98b257bced832042.json` (~11282 tok)
|
||||
- `00bf4ac62988ea0494c2ffda33b92affa91df5763081a54b4c7d6c308fec999e.json` (~269 tok)
|
||||
- `01181572b4bf0d966f8391007f85c7a58838d83ef521a9fb455e180c704c78be.json` (~4182 tok)
|
||||
- `014a5c34e9053581d48425c536c637ae0a2f48dbee99b69bbdf0e1fc716c3c10.json` (~652 tok)
|
||||
- `01592776d76cc70596361b4b3ca48531b3daea6674df45e2acaa0ad0a0a54cb2.json` (~13794 tok)
|
||||
- `015b2f7b89794af09e814aea5781aa51704ccb852db0b193f0f135a351831632.json` (~69745 tok)
|
||||
- `016845199833895f368a40f072564f7b5502cd652cef19f58dffb9c1931bd179.json` (~1092 tok)
|
||||
- `017525d533acff26d6ddd879e54b6284ce066a950e07536ed20b5b750b031219.json` (~1216 tok)
|
||||
- `01c91aa6fa18cc3129d3f6451a04b235c9528a88a165b1b6126f87c37bdc4048.json` (~14877 tok)
|
||||
- `01d5ae01a90dcbf2812ac05362927a7f9320cd5f20875838057caf559be8c5be.json` (~224 tok)
|
||||
- `01d75ef42f259e5752e9ee274fc1e00781b485b87349841c5c335ad92ba964b9.json` (~6308 tok)
|
||||
- `01e466b4c1a76bf9b35f06bea698dee8e27cafca61faa6985967c8e77b79cd10.json` (~269 tok)
|
||||
- `01ffefd3077e3989045ab89dbd8d9951d161531a0bef7cee8f652de34593ef79.json` (~40640 tok)
|
||||
- `0211c966a13950b059d40d37ff94e62d5b7ba25b554407097e4f964eaac39102.json` (~206 tok)
|
||||
- `023e107588bcbc8780b5fe90b346a96c5243e9192778aefb9bc9fcd92b5cf692.json` (~3708 tok)
|
||||
- `0297e5997050ad938795b7335ba2e08338aa3e4b36046f19b34c3069fd8d4e60.json` (~57184 tok)
|
||||
- `0378f813e93f2aad0de1964b16f2a399b4d64edfcba863e517414f4f1a262011.json` (~47142 tok)
|
||||
- `0399a9216133ac3750c1bb102a4d72efa9de2bd7624d52e4da1a072e0d767ad0.json` (~269 tok)
|
||||
- `03b4bc7e0fe5c91b086a3f122273dc82b760853a23e58e0b7f477d7d13b3f049.json` (~9331 tok)
|
||||
- `0401198757a993070c98c754996a960299d605d55ee524b2f1aad817fa94322a.json` (~3011 tok)
|
||||
- `0402d281826a30116fa7477996ab048b01a8375c95697e14b1a146e0138897e3.json` (~269 tok)
|
||||
- `0427a4a80364f93ee0a2d9991ae1051c4848345b8354d762cf72e2ba0dde43f9.json` (~28159 tok)
|
||||
- `0440846ef240f3abb98b1ca27304c5cdbe56779ba1cc857850dd7a26cad49b66.json` (~2935 tok)
|
||||
- `04583d91aa48af789b952136fb7c52041b0d632793246105f3af6fe070727a07.json` (~2049 tok)
|
||||
- `048866e1c729499c6bb7a67acf6872a9809730de276a43591cb6cde2de45acd8.json` (~44131 tok)
|
||||
- `04e3a10a165fc6dea955a31ec5c48db1e75c23545a74a734cb538d8ef8ac06ef.json` (~7004 tok)
|
||||
- `04fa04cc7831200204b6861ea8b8abac86e90920a8d9b0a8bdbd38e57901f18f.json` (~37240 tok)
|
||||
- `057d67f05183f6fe7fd3414cb6f46de234ebe58fa3dbca7dddc5441ee3f8a2f1.json` (~6218 tok)
|
||||
- `06a0b286e4c22fb4c981239820aea2042950d1d21dc96de40226b14cc696065e.json` (~8244 tok)
|
||||
- `06a49db12e21eff497c05a713d0cb6b1d8dcd7ab2a7fc336b10d9a5f2130270b.json` (~13794 tok)
|
||||
- `06b99babef4c7ab862aa487145fb2e6736245ef4ad565fe0e06a24898c3875d7.json` (~352 tok)
|
||||
- `06fa357013ada2065e2ea791488f988cc75fa423cb1ed66718dde55c14b07ba1.json` (~378 tok)
|
||||
- `07014154947b85f86499894c59be83a316bf3fc006c8fc55c316cb893462977e.json` (~3083 tok)
|
||||
- `0722fdc95d0e7367abdedd272ab781b00328a0a1a7d1adbcfad533a45f2921c2.json` (~323 tok)
|
||||
- `074a8bce08af84573c3e948b6dabbecdb6a7e99daca393ddaae4f816b1b24411.json` (~309 tok)
|
||||
- `074b4a288109008bdb35dccbcbc81a7e2b029bd17e4707fb967a94d962a77b69.json` (~4508 tok)
|
||||
- `0752d0d43dbdaac573d4c7129d228ad267a5a5d8bc6820fde4b6e6a6a3062100.json` (~586 tok)
|
||||
- `0757234001cdd8473e1edf6271d7f53808c84790b02b9f5df9da5f78ba447ca6.json` (~3592 tok)
|
||||
- `076584c4107b97fc37ee7957c3c616efb08c8e8f671a459f856a6301d267d0e2.json` (~14044 tok)
|
||||
- `077d2d06d6281efca08806f20dbbf41b4e4dc39af2069e261ebf97e80efd951d.json` (~81 tok)
|
||||
- `0799df8da46779b19ce58b3ae776614f647d6d5f009f418520eec8897f538f9d.json` (~1961 tok)
|
||||
- `07d0b1dbd63fc8b71cab7d602d56453c4ed0a4170b3efe8d387de29c09487ead.json` (~467 tok)
|
||||
- `082d2508e015e3002b8399b37a4562991cff2b2d747f1e800d370c1f3e88b028.json` (~5332 tok)
|
||||
- `0832cb54f1bb84f88c887f30f250279e8c8b3969a73e8e70917dff7a1dae78e6.json` (~652 tok)
|
||||
- `0857f9be25466f6f3c0da48438f7c7cf35d0140010f4639a673c2f35ae98a002.json` (~2468 tok)
|
||||
- `085fb3d5b8d5133be4bae28881658bcb2e5965798e6869b1975372b9d7d6bb7b.json` (~21866 tok)
|
||||
- `089246da170e288777e15eb8a6afea3097826290b31a9489f5b2c0a54658a4aa.json` (~79056 tok)
|
||||
- `0894c6309b9aae7fdb3b1aef7fb1dfaf1b55fd8f31ac96a8112916d55ebbedba.json` (~315 tok)
|
||||
- `0895960bce4ca499c4505de3f178508e1cd9dda69a3e8ab43195493957793a34.json` (~7103 tok)
|
||||
- `08c5e9e51057dd143336ebdaf2bd6af29481e79f95bb093f7eded2545bc29340.json` (~416 tok)
|
||||
- `08ef3af0d39f777dbc16b914fba71d5548fef9fdf88f31cf21620d70555fcf6c.json` (~78 tok)
|
||||
- `08fa451e66078016e0b2aa4c26635b6ada448d45d6d1c3d8ca9f0a9b53ca0593.json` (~336 tok)
|
||||
- `08fe84785dea915ebe3bccb75b305f1540b4fbc792fdfb2fb0bcee237164d6cd.json` (~6908 tok)
|
||||
- `0930d5c30f6238fc629bc312e105bf9d9400f85aec1ab3cada93bb05321b3b2d.json` (~37148 tok)
|
||||
- `093f1754670d5589744a31b6418adc392e01a959166455588f22822564144b3f.json` (~3601 tok)
|
||||
- `0944870ba2c03c7a45de293a8573c546ed6b0744b7c05134d9b7610e63636449.json` (~9949 tok)
|
||||
- `099d8954b4b123209ccf84667996aeddc044696f5219ca30afde0727ef8869da.json` (~5578 tok)
|
||||
- `09cb38261b447a3240b1361064d5bd8dada4c7eb8430b961c0476e902bc11ea0.json` (~8966 tok)
|
||||
- `09fc4c36a264a7b935bf7ffe9e8506fc3452b8e90c8ecb7e22d6a740854ebc76.json` (~1802 tok)
|
||||
- `0aa02adf8d8ad215f71ecbff28d24a2f1a3a889b3c9a27c142db458dc41a9eeb.json` (~652 tok)
|
||||
- `0aa0df0f12cecaa972e3c98e639d157283b73052fbf6caeed9c1429188e2c60a.json` (~20406 tok)
|
||||
- `0b48e2eca6f84770837387035ee5699a2c2bee2a9a2b65b8d1a01c56bc241368.json` (~13794 tok)
|
||||
- `0b4b61a3378363fccdf8bcbb84cf49adf909b131ed2cb415d540883b0d5b14cd.json` (~357 tok)
|
||||
- `0b84abe002c91c4608ec56b28a1a39a066c1d64f8bad5182c89a905e0f51138f.json` (~6400 tok)
|
||||
- `0b91b1fcfc8ae3bd247107dd81b84a03aea885632599033c06e8f9f372f8d967.json` (~504 tok)
|
||||
- `0be7df8682cb4642ffdfffc325b2525408c7db266ae8651ae0f600a01fb488a0.json` (~722 tok)
|
||||
- `0bf7323dd6f181b3c8536da6fda90904a6d3ea5fb99d9a9211175beadd33ae49.json` (~1973 tok)
|
||||
- `0bfddfa19374691c60a82a0208b83881a8c2c8adc9e888576e7aa64801be9f9b.json` (~4672 tok)
|
||||
- `0c3d78f91fd9dd44ca121663dc9c587c579a21feeb30d8c94dbf684da12f71b1.json` (~81 tok)
|
||||
- `0c86267402031c08bcc3b4b5a1202712295c728654b1958f36368c636c0416a0.json` (~3581 tok)
|
||||
- `0c8bd4d02d1ff7a1f62ce8e275db548f7944783cda074b1459f31c2353ec4f5f.json` (~42611 tok)
|
||||
- `0cb92b0a18aa5d0c783dcd326f75cc01ded87ac4a20b4cab2d938bde46b5ae8c.json` (~1392 tok)
|
||||
- `0ccb2e06c2139ef03aef6ba894efe1025c61daed9a9ef15b891319ecc4d9652e.json` (~5859 tok)
|
||||
- `0d723b61b4538aac8b6eeb6d7aab867f726bde72cef6576bbf873a6f5db41711.json` (~5459 tok)
|
||||
- `0d7b9090e2f9de46c367cff6e69cc8e7f2706f713987a883ed0b697fc7df2d88.json` (~231 tok)
|
||||
- `0db57ca4ac0c3ee6abc97242d0a04f274b18b83ba32d3b9c2b8c39b949c52fba.json` (~2878 tok)
|
||||
- `0dcea2652aaf077ffb9d6f61c387ac6dc42e03d347a5ea9dfde069c107ebabe4.json` (~26116 tok)
|
||||
- `0de27f108310a09dc43627f38940985e713743002cf2c03fd0dacdd3e9c10c75.json` (~6565 tok)
|
||||
- `0e14b548fb2b4f03553bf5e3eace9d20c5370887408f3f2affaf4c569b36fe2e.json` (~14044 tok)
|
||||
- `0e688ceec91f4b7c192ec781324e9ddaa517ac653dc2256095ec17c49cb12928.json` (~309 tok)
|
||||
- `0e6cbe399873548fe0ec938c9cf3ee70b617b329dcc630b4dd7f1490f343e756.json` (~13343 tok)
|
||||
- `0eada2ef529a3511cb04b120cb3d2247c7822f9aa4c4d9dab4f7cfdc72d31e8a.json` (~1980 tok)
|
||||
- `0ee129793c23e5933eade7198c1f08c5d84c9c26139e7eb7ff067d5c52d08d1e.json` (~6165 tok)
|
||||
- `0efdb7988a262995018cbc7009c3ec35246c85602f7a6182cec1852b8c765ba2.json` (~859 tok)
|
||||
- `0f5341ff6ca93ba2938f8f8745a9739a2f5da1705bdb8b6abca03899d71e75c6.json` (~224 tok)
|
||||
- `0f9a00a3d6d2161c402e45a68e7de9286018427a74f5932f18d96041a1d44cb5.json` (~6562 tok)
|
||||
- `0fa8ec51434b17edc5a26d6a037f6799618b873ebae92d6eac751f0f3578c038.json` (~2079 tok)
|
||||
- `0fb70f67355c6ee9bbb1dd9f5d55d21fdd019790f551898c33e6dd97d0742331.json` (~468 tok)
|
||||
- `100b50bbfd196f55f7a1ea757de572eff68aa97bb88cb2dcc13d16f23bbe8c0b.json` (~80 tok)
|
||||
- `103e27126a9e5950aa8c8063c54df342d6737d340c5411cf667b24a00dcc28a1.json` (~3251 tok)
|
||||
- `109127319f5fb36ebd4e47de8487b09aa4338bec7c17117d77cfa643c9177ee5.json` (~368 tok)
|
||||
- `10bdb81aeecaac694b2c297964b42c596e5ab715e2f71a7e1ea4c069a0e09be8.json` (~1122 tok)
|
||||
- `111178800ae1876a431e527f3e0211c41e066afcf3636483674e1de0177fe726.json` (~3496 tok)
|
||||
- `1115d59cae0de03be0ebfeec800e4bc7a3142b1df07c22537ff00164f6d4becd.json` (~845 tok)
|
||||
- `11313de7f08ec2177d21ded84e4f37426d6bc50be2d1fbc729b4ee73fcff6b46.json` (~42896 tok)
|
||||
- `114f295685c5796b4dcd914d8c84d0e74823a3000bce5dbf796f52519176b793.json` (~275 tok)
|
||||
- `1164b6d75143d4caf1a8c3ccd18258c793607dae4d869a5490fc09564b26c2af.json` (~214 tok)
|
||||
- `1196f6d0f42ffbd96922026767a312e1d25869970da006eda64602bb96691c22.json` (~228 tok)
|
||||
- `11a454492a215c5707704555427cc5a8a8d8c9e37140369fe3fd48964b6cd6be.json` (~3120 tok)
|
||||
- `1221d9acb10eff7e1b0ee12d8ad0299cb196b7f4a9c3963819f227c0bb8a26cc.json` (~269 tok)
|
||||
- `124ea916443ccf66314b5ccf8b768a2d0156f6c88966aa61065e505f2fcb8ba1.json` (~13699 tok)
|
||||
- `12c733acf3d86ce393643e971e1f4940fd1758a2696817d2f2a4247f41b2f480.json` (~5120 tok)
|
||||
- `1312e11eefb522023a6437df3bef6b7a618dc2a482005da8315b46a9fe0264a6.json` (~7909 tok)
|
||||
- `1317fedc2a67f4453983d16264915c41c2a2890b5a76b4f708c391e56c20803d.json` (~5591 tok)
|
||||
- `1318036e2902ba5c87e3abfba1d7fc3680c2dcc5a6fe04c957e5b147cb26fd4c.json` (~9139 tok)
|
||||
- `135ad5694940e707ec7bda78857f07194b8d507860b55af456aed9f29b4a700b.json` (~269 tok)
|
||||
- `1360f9fdd34c9705242878cc84f73f7fecc97ef1df46fc7ac09477f4e15aaa7e.json` (~37148 tok)
|
||||
- `13e2ff80a35b1dd9978c0ef2ea8d55f7a2b31a8f795a922d5fa830f33de23ea2.json` (~7716 tok)
|
||||
- `140aff7f6dd37516b4ef6f3ca626f82f50cfe71e4f3a578be699a8ef72fb01e0.json` (~13687 tok)
|
||||
- `1459add9026c6903d390de90c7d1dea4d5a9d381a3930785e7a480dc27b5ec00.json` (~6672 tok)
|
||||
- `147dd164653fa29be6b640dbe5bf13d8a9e91df0f7275ac5a86be7568216d457.json` (~81 tok)
|
||||
- `149a262f7c5d8f2cee31c3719e56f78545b5897d591c51f9cdd916c56e035998.json` (~1317 tok)
|
||||
- `14a93f201d19d457092b9bf881e8ac62f44caf42d70192b2106944c8380374d3.json` (~76 tok)
|
||||
- `151a099eec558a9e4c89bc08aae88ba096681971be1a94c7625343fd9a3a88e5.json` (~7640 tok)
|
||||
- `15224225923ce6697e098312e1dc8429b2768b415534886f344c212446813d83.json` (~495 tok)
|
||||
- `152d9c2445a8247d38b52f6ecab0cf6f36ba144aeb93e0bc7603445e7ed2c6cc.json` (~5075 tok)
|
||||
- `152f18c90cd608c8a87b9a2569790aa32bb0684da62ddd22284b2efc477c58ab.json` (~9949 tok)
|
||||
- `157c7929f5a2274e2f98c623d21343ab2f01711d4fd4e742caef92e2bf2b3463.json` (~4857 tok)
|
||||
- `15a2c5ecb0fccd8b13c841d488ee0a3ef8a753347864ac6748b02cb48bdb3338.json` (~9101 tok)
|
||||
- `15cc2d188fbb7c7bc937a39311efdc8cd2536b01947309e45f5563c0e40a7923.json` (~1376 tok)
|
||||
- `15f43122fe941435ef4d1c92bf796b0198998b92df0b915e6e173a4a0d3e0dcd.json` (~2943 tok)
|
||||
- `15f7e2e59dba0b3c91ca6a2c12b934bcf6a1f7d75a13c4aeb0d403dcf9fc95e5.json` (~19624 tok)
|
||||
- `16307f083b22977db26185b0f7407bc40e4f74e2098040ae978316d99d4444d8.json` (~335 tok)
|
||||
- `165d0b653a3b68d52594e151b8e06e0fe93b241b56fcda872811592eb2ffbf4c.json` (~40562 tok)
|
||||
- `16853a21974696cd7ac62e3b2c1aecf0dd561d82457f43f1d92e404231caef07.json` (~269 tok)
|
||||
- `168853e90d6235d9054ea90422d3afd1189be5b49efd2a35a342c71fb347140e.json` (~652 tok)
|
||||
- `169eadbfd764f364c5b3ecb27f3112d552e98c1bdd4e0ae416ed5f821cb0e6cf.json` (~13794 tok)
|
||||
- `16c33a7e7d38cb29ec2063d0e872d977ea8e2d81bb94e0afe95ca823c6eedd0f.json` (~351 tok)
|
||||
- `16ff7e6eeb5702d2ffacc9e210eed5325524fa724aedd15ae449e52a14ff9e01.json` (~88 tok)
|
||||
- `1730cffbec8dff2313f1e1f655b1026f93c037c722cb0198f39d2c15da08ee60.json` (~1992 tok)
|
||||
- `179a0d9514a4bfe3e69a07482e79899fa08905676b1b5f8f9b1f9d0ce55df3ad.json` (~269 tok)
|
||||
- `17c6bf76cfaf2951c6567ecaba00b057c5bf1a8c380a669df83e1a3cbecf630f.json` (~482 tok)
|
||||
- `1814c8441ce6baf90c83e273914a1f472136a2e348e12e9eaf902367a650574c.json` (~323 tok)
|
||||
- `181955f831ace827cab57775ab5c606ca3546d23e3b084d7bb3a386ab30603dc.json` (~182 tok)
|
||||
- `1870fbb4d9d0c1c96f6d0206b980cc4acaa128a432b581bcacec7f6fe866eecd.json` (~1988 tok)
|
||||
- `1880ee75786ce6a428cdaa0f911f58973080482a358b7886fb6aee65c43beda1.json` (~2444 tok)
|
||||
- `18dc5928206967173c007cbb765877d41488abd0fcb0de1d56abecbaf3f724b7.json` (~9240 tok)
|
||||
- `190d71d1c4df9c6394d58d7e36010f7614f4143a0d91d41002e0f2467072fe8d.json` (~1064 tok)
|
||||
- `19134f041b4f3b6c95a2149e12e5c58ebddbff604d739bbef6226eb1687d0e34.json` (~8959 tok)
|
||||
- `191b4446d235d84fd3681d7e952ad2552fe572d3a5757a5074074549c4e27d50.json` (~1829 tok)
|
||||
- `19526896e2fa23848fb2da597198d3a3717516be1e4a527ed21ff3d835c63fb4.json` (~30722 tok)
|
||||
- `195ba5f52572d4e963afc4d2d690a1ff166ae35017e308f8a0ed4a16642a4316.json` (~9746 tok)
|
||||
- `1968aef428601da0e4b23250d1ec6c3d1533630b4d0914f6c70f4e387a59728d.json` (~1990 tok)
|
||||
- `19bab18f4a22b0290f91cbd4de58488fe941677de1fb92197dc1c4410a878feb.json` (~7440 tok)
|
||||
- `19c8003bcc5314fd08d9000b104fd6d443d7959d5babc40fe8b7eb1397056352.json` (~2145 tok)
|
||||
- `19d35b46440f9c21e87df58e15785f9aa35f403fb4274c0acd930c47899273a3.json` (~82185 tok)
|
||||
- `19d6cd319d6a3ecab2357d41a7d26db9a7e35401b0c3e1dd87dc099b71429728.json` (~2283 tok)
|
||||
- `19d8fb8222b609932635a102044f42a193782e18e5ec575a71312ee87bb2f659.json` (~13687 tok)
|
||||
- `19ef06be285e1769d58acf0223c5a445d872c31d629d30dee1c5eeb06c69ded7.json` (~35085 tok)
|
||||
- `19f40018949a73961f1be70427bfa744ba1dbb6c67e597c65b3f652d357be8f0.json` (~324 tok)
|
||||
- `19f57f6943301a280db1ea82fdca9d564ff52d97111d7341fc788573dad0a7aa.json` (~81 tok)
|
||||
- `1a0f0b246ea02f1e6d7196a83af4366a44c5841bb907e8edbc44c30920ce342b.json` (~13750 tok)
|
||||
- `1a91351d50722bca9da67761028884679cbc357a5587e783ce691d7cab7cf65b.json` (~1075 tok)
|
||||
- `1ad8474a865962b24820ff0d953482ce1a9644e667ba136752e0388d24f0ebf9.json` (~4490 tok)
|
||||
- `1b3eb8209fe8ae83a46fe284c12c8577df135d756ac662156e4a61eab79e5601.json` (~1934 tok)
|
||||
- `1b5a6250958790565da71172c0f2da9c26e4651aaec7a353d34f33555f38f4c2.json` (~1904 tok)
|
||||
- `1bb7af2ce570f59f08babf055ac4b6599c1f138fd580726f6f8056071d6790e8.json` (~76 tok)
|
||||
- `1b8bc9a85e6770511db55fe3c159e4333675ff5c39336c212b86c7c3d84d9c76.json` (~238 tok)
|
||||
- `1bce3e8bcdfc4ecf26ebbc435d3a25b2d7d00276dce7874fa87de41f72248b00.json` (~1082 tok)
|
||||
- `1c0cfaeda41c83a55365b1b5b340cf80581217b083b93ce0dd156fc886276256.json` (~3131 tok)
|
||||
- `1c85f27fe659a6c257eeb5157b01c9b454293a22bc14be2be6c45aaefecd1967.json` (~753 tok)
|
||||
- `1c906780916fd91e042adaa801f385dc55bc1f29c851908ad61f8c5db0040107.json` (~2163 tok)
|
||||
- `1c9619334c216e67eec7d7a1c420d26476001173e926088e46d4d2acdd52f8d4.json` (~320 tok)
|
||||
- `1c96714a7cfa6cd6a3494d1325905a74a6caf801e63a5abaf2a20bb8c8a9e3bc.json` (~16210 tok)
|
||||
- `1cff99e0f708b2a718e7703d855083b777015610574256103fb398084f05493f.json` (~8989 tok)
|
||||
- `1d233ed5870475c31d523f30699006ed5516c7557bcebef9488dc179ff83c152.json` (~1122 tok)
|
||||
- `1d45d90ec7a9bb9ea8b74d81c892b24013a9c2d4499472bf30b6dde0cd2f6190.json` (~9331 tok)
|
||||
- `1d490c507ec55945474495705d4ef2ccea924f616bb43256bd143818c1ff0007.json` (~13361 tok)
|
||||
- `1d6df2069938454141daef26c8ed09b7885725a4598541270f941bf8fc33a465.json` (~4675 tok)
|
||||
- `1d8fe74b05acd86ca281750b4632fbf2f597b035b94b0de94e27ad046b756778.json` (~15817 tok)
|
||||
- `1de44a0c73b0a9fdbb5496796c1478a01baaa8de24d4316746ad4c72b577d01d.json` (~13750 tok)
|
||||
- `1e2f5a8904c7fd7c4e14f5b6c9f7bac9fe6bcecf73c23443e125c2d8966091eb.json` (~13687 tok)
|
||||
- `1ed0b15ae075c862b918b1e937d53da379b474d8a9223ca117cb68de41f7a47f.json` (~1317 tok)
|
||||
- `1ed8551d553792e11c27e1bb56eb402f74c731f40bf38b672dcf43a5b847d426.json` (~73 tok)
|
||||
- `1ed92f3d08961cb53cd981b7a42d9c0352078956ab010c4179498e7b940fba62.json` (~9746 tok)
|
||||
- `1ef5dd882e1ea3b6ed1329ff0047d21ee2be8340c12e537b57ff2fed1278bd4a.json` (~9756 tok)
|
||||
- `1ef77ec32c809ea9f6abe455804f5630dc3a9693e633b03068f3546dfeeab099.json` (~3149 tok)
|
||||
- `1ef8a1c88f58d9a1f913f376629dbc7421fbb4f6d0c68e777f46873652df204c.json` (~39527 tok)
|
||||
- `1f6c6535ebbdcedd95bf44e6b96f07f1234d7ea54cb747d6e113f90ccf4bb7ea.json` (~13910 tok)
|
||||
- `1f7e683af7e73fefdcf14ce40c2ee81dfa8aa113b7aa90b51067d870710baa7a.json` (~332 tok)
|
||||
- `1f96efaa663aa96e3efc983e62dec6bf5c041fe09edc8d76e9d82344763e01ff.json` (~44715 tok)
|
||||
- `1faa6f424fab9df585d0b42dc837a4e7b7e8956994c3eb83a12e8ff867ff49a3.json` (~76 tok)
|
||||
- `1fe907805f188b6cf4e490bbab545a31e875b83a02bb0e7220e1b8666ddee3e6.json` (~332 tok)
|
||||
- `200ce2e569a9fdf32eccd599d8d7481cc255050ab65f74b34c0eaa8a071bc525.json` (~8959 tok)
|
||||
- `203a5b806478b8ffbfad1b9f0e6d211c8d8cb1fa79b42e26db193aab4f88b0cb.json` (~43662 tok)
|
||||
- `204755bab645cbf0ae12bb0bf8787de41a1f94e10e2988eff7357b93085e7466.json` (~8392 tok)
|
||||
- `20747f7fabc3eefc90e2ad0cb4957ac48d975786d8eecbb334b888c012f28968.json` (~81 tok)
|
||||
- `20a75b0b19d807e6397245ac948f01fc48666a82513faba2a1137762b28c6e3c.json` (~269 tok)
|
||||
- `20ae8b9f3604f491781f98bddc1122f40129f326f81f19828f87fafa94629a23.json` (~42348 tok)
|
||||
- `20f4f40d81b0a264990d7aab7acdf82a1b66fb32d803b2b878f056973584921a.json` (~474 tok)
|
||||
- `212272920267eaed9f41bb33936ef767250734cbb810306b712126256ad9e94d.json` (~351 tok)
|
||||
- `212d334f64ec0ad5a158d6bbb8752b96f7efbc2b4baa556a1f2e393a10a7628b.json` (~368 tok)
|
||||
- `213c3cfae0a856a1f1c11f8a08d19d2d4d3f2be6eda2e1c05518b3687895a55f.json` (~269 tok)
|
||||
- `2155a5a1575dcef9f688750a7ab5aef21abf6857b99ca64f0140ec2d95679d8a.json` (~464 tok)
|
||||
- `21646b4f891df0a3a87bbf8bc22fbed445cf2865d4fa7c6b1c3366b5b5a55b37.json` (~1737 tok)
|
||||
- `2191b94d54667aae003b310d3d3a7f1d8b8dfc17b141412e7db8d73be4d905a6.json` (~40335 tok)
|
||||
- `21b020e96fb39c7dc4f05e58b046551b0434adcb3fb196276a019175b1d189dd.json` (~4422 tok)
|
||||
- `21cce875e47827e8e2a8c3509265db54eebd3662ece3233823b6dc4c23812066.json` (~368 tok)
|
||||
- `2205e0519599088ede54092b2c13d182833ad49f15a64411c26758d1cbcc30e4.json` (~6577 tok)
|
||||
- `2218f88cc182f9064384f7ef7a3ac49908b837dd2d067cd8dbced90e8d043884.json` (~8526 tok)
|
||||
- `222393034308f06b0a75d7a9005d69878a1d47b9e8644df31a164bdff7ffd4af.json` (~37672 tok)
|
||||
- `2227b13869cd4b0fab8cf5f9725b140ee12668f2bfa111f5a78240f7be5b463a.json` (~1819 tok)
|
||||
- `22327e60fd127a993df2718d2be9464bf609a9716a5c09c00203a8351ab7b89d.json` (~2759 tok)
|
||||
- `226d09a3ccfa62cc05d1b61facad2fd5f269e9f41888caa8457f1dc846435b4e.json` (~3797 tok)
|
||||
- `2272750056f162f7c74228b5a1cf2324b1299841d24f98d0dd31696682e18f28.json` (~2727 tok)
|
||||
- `22ceb8888dd8b8d541e7de6dd276e900bf1e38a76202abd19322f8499b093edd.json` (~368 tok)
|
||||
- `22f1703655d94d51eeaf40c45b184283301dfe0cf0ca223e53736822c655f428.json` (~9949 tok)
|
||||
- `236dbc464b50c074e467d53cab78fb8af06c934b191349a006f12477317ddb91.json` (~637 tok)
|
||||
- `237f88b4759f44668ce8cd4941b4231ee77fcb94c932ef3f283e596de3de41af.json` (~1262 tok)
|
||||
- `23ae45376874ad36b52972b9b9031b66aa76f34d1de4b793801908d2dc0bdbc4.json` (~4128 tok)
|
||||
- `23b8085ed43aaaed2d876c21ecb494fa28af40a1cefaffcf5ffc4b131d9ed3f8.json` (~1662 tok)
|
||||
- `23c395989dcf7b918529179270f64198321dac7e928f32eae73de0d273ea677e.json` (~13750 tok)
|
||||
- `23cbeb01ae9cd9669fae81ddb2b2d6c7f89abd17b12365a5f6b256a32a717616.json` (~490 tok)
|
||||
- `241863e2714230e0a277eb757d3e7b8e7d2bf7ae4b52fca140fa70c922b5d63d.json` (~15467 tok)
|
||||
- `242faeda0abe77b1e66a515917a267528fe2d1b0c091dbced56e493e9cf69737.json` (~3934 tok)
|
||||
- `2465665f0153b8f74692bc9dc35d043f715c536aed7d2e5f04b54e8a24433c1c.json` (~12466 tok)
|
||||
- `24829855230378343e117d2d2831a59dcb4efc0c8107a76984172094d819a306.json` (~1196 tok)
|
||||
- `24ab726fd0c91d0410376503f56b1cf874ba7f2793f1b91fe5d6d1356104f033.json` (~13581 tok)
|
||||
- `24c50f06d81ff1284c8386cf867cc7d96d066a502dbe0c48b87e964d6877fdf0.json` (~652 tok)
|
||||
- `24e1456077e04d7c734decf96aa9c9b3c8e5b75e0bcfc274e747c6ed65c93c0a.json` (~13794 tok)
|
||||
- `24fa3f5b3b5b6d59088387aad4130a5e2d657f88706205d6dbe57e8afda2e425.json` (~722 tok)
|
||||
- `251ce70fbb55f062faef6259731b4c82238f4f7af22d2c1ed266c86c15263b99.json` (~502 tok)
|
||||
- `25389b2042efd3ef5da1211191cd0c53e4d697dc319451cd5c7289fe6891dda0.json` (~1946 tok)
|
||||
- `255ff40d78e624e0f8c4e96ee7901bec920e078d9960cf9cdafa2a97453e877e.json` (~359 tok)
|
||||
- `25d9960f7189fbb39fe69da1e664f8142f3aa9df8bbccc2253e148b2cfcfade9.json` (~269 tok)
|
||||
- `25f6c2a43106baedfadd95c63a6756af97347a44dec2331761762e474aae4efa.json` (~2638 tok)
|
||||
- `267c67c5b1f429f43ed99481e42652c822ffcf51d0d0c6ba4121d120c9f74e4c.json` (~441 tok)
|
||||
- `2729cd44ff4befe72984d232b7ae9d0a2bda3c0ad487f7907c73680caec307e4.json` (~14268 tok)
|
||||
- `28d052ea57fd7eef39d0ee9fe1d7c7531b0ddea082a123a6997c50f0703f2d98.json` (~1340 tok)
|
||||
- `2930482f7a15ac6d50037b6465a084fdc9cbed52609068751020b5c3016088c3.json` (~1279 tok)
|
||||
- `29cd8d6bcb6dadf0782652207d01f8eb0435d068aaa91c582d98a961fe3ad8fc.json` (~2251 tok)
|
||||
- `29f9d3dbb7661a1fec506165923e621ed0a0c0f1fdab925cb1759bdaf50e5f03.json` (~154 tok)
|
||||
- `2a2bb41a3bfba155bd19f74ab5cf8ed888210b7ecc7a11b5be1dd100774bc897.json` (~2825 tok)
|
||||
- `2a7e32f31ea531530978500ba347d3eb463270ec6c4f4b0e300c84a9fca4729c.json` (~1131 tok)
|
||||
- `2ad72e7a427f8046c048381f2292e12f34926b58456a02043b407d6cf490a77a.json` (~28558 tok)
|
||||
- `2b0b0f1eb49df36a2e8fd4f88317124dd1edf9a0f08545a1a53accd6b47b16b1.json` (~309 tok)
|
||||
- `2cbca2b1bf0345399a27e19131777f4d537a479e5d9eea2aefe60bfa6f0c848d.json` (~5279 tok)
|
||||
- `2ce08b7c8b08e1091ac5949fb66ae92fa683e1be76fbaf7b41b958dc151441e8.json` (~24408 tok)
|
||||
- `2d59df43c2cc4d09ea8566c4a94d70cf1a59b8d86e8fc4f9d7c49e92fd356d75.json` (~702 tok)
|
||||
- `2d7bca77beedd8fb1f113d009148dbc494da3f2c7db1546b50716ad5a2008ceb.json` (~268 tok)
|
||||
- `2efd2ac807fbaf51e1d509f8703b3a911bd6a2e99f11b7738f040cd777e02232.json` (~565 tok)
|
||||
- `2f253d9538cf47faa19555fcece49f789b8e0379c5d654a0fb17c2b9037d7e6e.json` (~619 tok)
|
||||
- `2f314917390ab9fc2f54f98e76033a35b0ac32f11d28a23c6cef9b10c03440d5.json` (~296 tok)
|
||||
- `30e46776a7b0b61fbf25b89f2cbfc05cce0db668f4e79ff9e2b5094e3b968fb9.json` (~2029 tok)
|
||||
- `31925f74354626a58d815d51a97d505725f05f1cdd757f6d07c35080d040c3b6.json` (~10147 tok)
|
||||
- `31e922879882c5b3f25d90a04b60bd1d7379d4c9129844b17a6829bc4bf89615.json` (~2210 tok)
|
||||
- `323b81424dba12471c44d1dc784633bf1d5a69a98fe31afafbdc964b63dd28ed.json` (~7914 tok)
|
||||
- `324d5a22da8a2bc8a6a657073dcd1ca83f95aafde5553a4425c41fdfd18b1502.json` (~844 tok)
|
||||
- `32526ddea34efdc2b69417295f72211dd5610b79379991f1bacc52a8e41e2bc8.json` (~5167 tok)
|
||||
- `32a73e47aacef8b9773844496f1bbd6068795cb2532775f0118d3147fc8f19e3.json` (~591 tok)
|
||||
- `32b58522b3e3c7218eeed3098b6c5c4f877195b74425407770722c3dd4e2aa8e.json` (~1988 tok)
|
||||
- `3345b676a5c6e6c2392fc43615a0e0da1824dd13c83d3598598690f8c63697ef.json` (~15396 tok)
|
||||
- `3399458fe3fc893d4f104368ef5ecd4afb95871b46436b2d09d9c71e700cefd8.json` (~785 tok)
|
||||
- `34ec637d899925d6db1ba12bc8bf368183f2b0e3636ba7212b2056bf7bd642c6.json` (~85 tok)
|
||||
- `35084efafb380fce518784bf2d8b13c3683acabd3190810991b97a34c52418c2.json` (~396 tok)
|
||||
- `350898ae8fe9d6a099d42d9d292c43efb8a1a37182085f7acadb6b34602d6024.json` (~11199 tok)
|
||||
- `351d8046e5f9f289979f6864f99509fc2fc25b015c2ad343ab4b99193132e56e.json` (~715 tok)
|
||||
- `35687c768af0e0ff77d76bd1dcc018cd30b6b20c597a67586137ed0a9878bd61.json` (~965 tok)
|
||||
- `35ec72e792b59b19044577eb54a5e7540f026ac35c6003306f597a5f1387d227.json` (~5448 tok)
|
||||
- `35f823871a12ed1809e20d70a663c8a8929fb35f3748705499e32df6cc4ec5c6.json` (~277 tok)
|
||||
- `35fd2a3ceb004b1d47fe5ccc509be97f4d380faebcfff32399237e9dfab59c00.json` (~6584 tok)
|
||||
- `367e004bbee4bd3a32e05eaf817f2dae69e39266c36b1649e4d3929362e383da.json` (~930 tok)
|
||||
- `3702a2af17c73eed0470df46cecddc0b4c9a93aaab893ff5292d5c5b9b4ed512.json` (~9636 tok)
|
||||
- `3714706b1961d4945195e8613ba7a5fcbf550c040e671d5860f2e8d747dc477c.json` (~6708 tok)
|
||||
- `37a543436eff088729eaa2a790c9ef9e58f59f226122e12aa2a0e6c38bc6f5a0.json` (~1870 tok)
|
||||
- `37cfdb1b9139384b7afd6b71a19606306bde221b63a2c4b3eb65ec5307855e45.json` (~17007 tok)
|
||||
- `37d60444e71080df18fa511b971b5c8d3c62ac5e8cf74527dcdab21a2ae69a76.json` (~561 tok)
|
||||
- `3816d68655555e3b0fbaa3474816b59e27e52bb3b685686470a4d504053e203a.json` (~1110 tok)
|
||||
- `381ebaece388d903a5945752e2c6a1b923416e6e1f5373b13b903bc69639919a.json` (~6344 tok)
|
||||
- `386ae1e8bf78901053b32e9e09acab3017e1b15c172bdb9a7f0ba752ff25583d.json` (~586 tok)
|
||||
- `388dd55aa581a70dbce06969cd866d8f46cdfaa55cbe211302cf06acdc9fd64e.json` (~2999 tok)
|
||||
- `396e466ffa3cc396b083d8500b44a94b3d48f18866394c5d7cc8d1e2488bfc13.json` (~801 tok)
|
||||
- `3993a289fb925bac9ad356f94888ff857a44d592cf7d4c1815c3702493f24568.json` (~9385 tok)
|
||||
- `3a1c0a4d790a5bb1e032f9fb3e075ae112f12ebdcf0d096e53d65eddb606120b.json` (~300 tok)
|
||||
- `3a5d7638d9686a803cdaf98d5696e7dce79ce8876e7dad5a4f4a9cd9b26f473b.json` (~14751 tok)
|
||||
- `3b3d41a9d9e7bf1a198634ee72bea3a02b87747fa073455ea33e203dca82b5f8.json` (~3990 tok)
|
||||
- `3b69e25c707eab4010cdbda8e2318a464bea4be6a2afbd7a985f7ea7706e09d6.json` (~2171 tok)
|
||||
- `3bc0e6a7a00dd29f382b9c836dbb598929fd4e4edb1b450959d00c8e4712bd8e.json` (~636 tok)
|
||||
- `3c10b54471efe824f1a83f3850f6bcca4e2d77c61410b1bc617461546a792a27.json` (~64512 tok)
|
||||
- `3c3cccc9607f4041ae193d3d0447f495c08dfa09f28d9ff75b4f73b98264d0d3.json` (~725 tok)
|
||||
- `3c576da7137df22cc5fbd982fd231d9d40d374a77c7d618081ba7018f5eec36c.json` (~4654 tok)
|
||||
- `3d7e3a102cd4fc3bc27f599364f8f63ec75e6cb1ea3c47bcacc0e1261b485a15.json` (~3783 tok)
|
||||
- `3da9ee357a09d89a26df59bd0393606349c2d0849078091078924f60b5cdfd51.json` (~495 tok)
|
||||
- `3db8027a5cdc79633520de23a59b83fb76ecfba3132ba8a4f52e785220157e70.json` (~336 tok)
|
||||
- `3ddf0434ea5d953052cfdb1597b14704a3692537f86a9fc442ce21024160eb1a.json` (~30420 tok)
|
||||
- `3de5f25efab468345f714829408944025550d1056ba2b46fa3f3f29bcd5a9573.json` (~2258 tok)
|
||||
- `3e4c4692b6473b063999f972a2cf88d768446bb245fd18a91f83d4ced2f1eaa5.json` (~847 tok)
|
||||
- `3ebb59da3bec8cfea2f9a426717e513a696eac7f6e08672e0a2d7f72ac811ba6.json` (~23800 tok)
|
||||
- `3f88bc2442422df4bf6036c3b2a1b3409779cbbeff5df7fb2c031bd14f973119.json` (~6809 tok)
|
||||
- `3f9cade10a0070913a41048bc2462ee7a4680399174c61c30c8d6e8a6e722cee.json` (~4780 tok)
|
||||
- `40138b45926e46347669dbe289bb2ea7597aedc0eb558bd0d39aacfef5df62e1.json` (~2333 tok)
|
||||
- `401cd381093f599bbb1439aed5ed4e0b4b1fd4026dea54a11e953bafc7289301.json` (~6490 tok)
|
||||
- `40a5802d6063d7dceabcd49010eac965e03377546e69b2839ce06e3bc19fb99b.json` (~3592 tok)
|
||||
- `41bc353adb8853621ebd82fa735c038f98fe3a7e14d44a2eba6a50f01b4cc944.json` (~1511 tok)
|
||||
- `41e4d75d97d06c3f52e77585e5d114ccd5a9ac8e21767a2b7c74cc568dc7b5b0.json` (~298 tok)
|
||||
- `4244574967e1be57fe35ab1b68d8b31aa262d50a728d8877e7af61f9dec09cd5.json` (~526 tok)
|
||||
- `42fc26ddae39455e27588e5cd8b7a61961a4e5ce52a4e6c77fce30fd813ebe38.json` (~465 tok)
|
||||
- `431735fb57e4e50e282f8cc57b09d553ef1b88f135996cfd253bfac5ff186bd3.json` (~2752 tok)
|
||||
- `43ba72a48acad7a07213454f6c1f78abccf9dd5ecf463d5863b83774dd99bf49.json` (~1010 tok)
|
||||
- `43cca59d292abf40819e07f2859c2dce727ca71a7334ce8384e9a1c50d922337.json` (~3604 tok)
|
||||
- `44245055f1f2aebb7db03f3ef508b06f9357cc6f5671308e8bfcaadf913972ad.json` (~2335 tok)
|
||||
- `44b45ed14c23e25aad3d891047d8f407eb08684458d1850b31e75380bff306e5.json` (~2902 tok)
|
||||
- `44d133b11103df5a5455f0e124056054065a43a4cf9bf6608a157d98a8247919.json` (~2414 tok)
|
||||
- `44eacc4abbec955dd4031ac82e902ced758e0fc319af2c17d2cc650d7c8701d3.json` (~648 tok)
|
||||
- `45a3f01bceef855fff2282e1e2f5a96edaf6e49eb2dd6774de2c951190939476.json` (~876 tok)
|
||||
- `45b2440b66a988758fb836ac379113401f941ef6a28b9eda47de0ec5b15d8775.json` (~2613 tok)
|
||||
- `468a56306bfbb4b7f8f052404e6f4d239bde83c10b206520cb8e9fd6631647c4.json` (~1038 tok)
|
||||
- `46936120f3fc1bfe47072b25b2af1a816c418616c4754b04053995ea6ba7e850.json` (~24480 tok)
|
||||
- `480dbaa39314c68c3d4da3c1cb184846dc98ea3f672516afb1683ad674bc88cf.json` (~678 tok)
|
||||
- `4864f62b6a1943ef81d8e4f903830c54ca1451bf8656bbcacf0b603e9ca39af9.json` (~238 tok)
|
||||
- `48e2a62af809b08d550c4c38d6312ef86dddbab71aa13f2b6058f83da4c364eb.json` (~1226 tok)
|
||||
- `48f612d06f7f9ac59bc69ee5cd647248f72809eb7cb81546a1d7fefc9a0c560d.json` (~16313 tok)
|
||||
- `48fabd0bf158cc4c2454a0f8feac054b1bdfaf3487e7a59ef8333589996833a3.json` (~27880 tok)
|
||||
- `493d95ae6177d34856630ff36929d3a77bdfdda78cdd13a3079af121c61a28ef.json` (~412 tok)
|
||||
- `495102d8b31e92493be8301b6d6af618f217d67f5d23eb73970ed03e4e6aae38.json` (~2679 tok)
|
||||
- `496de07d7e51cef1389ef84f25304ca54e6ed1d409b90c924d73cbf88230066d.json` (~49671 tok)
|
||||
- `4a16de26cbf3a4184dc0761323b9a08875225fe03c549d151fbc746db1144a7b.json` (~5507 tok)
|
||||
- `4a1af2b057f91ffb734e4dd1c6e02a70f7054d58da2611f6ff04b74883f2c46d.json` (~5486 tok)
|
||||
- `4a4287dcf690e9f19f6452e52c82efe2200dcc1c040b13272c606ee0df281c6e.json` (~50441 tok)
|
||||
- `4a48ff8b5aa860c94ce2439538aaf0a877bb759fe3f312b9a3be65dd8d121b47.json` (~3390 tok)
|
||||
- `4a5cfc964b051c80cf77dab98611bc81ee2af457f1fe2fcee99fec744ba1db6e.json` (~5301 tok)
|
||||
- `4a8450b30e8d556a37680a470ee5bfc9d223c06eff6dee6cbcd7b3b4a402b1d5.json` (~1551 tok)
|
||||
- `4b1e8b3876352a607a2d7f8945f9770d648a45f3285f916ab81b29798e1038b9.json` (~5468 tok)
|
||||
- `4b648434f378dc77cabcf7ad726d40ab8a3ecd4bd96c25fe2b9b303713b85cfd.json` (~71366 tok)
|
||||
- `4bf3a6240c3116e762e6c911280d0bc5e14d68dbd7b82caf61f8695a11587e72.json` (~1277 tok)
|
||||
- `4c776cf52b544a88c4c9a52d25dd1520c7a30cec47d7e0f08f0acb0eab7348cb.json` (~20902 tok)
|
||||
- `4c8845dd0c079a771cb7e6ff15708701de63a89516b79a1ec9a9cc9a2a5a4801.json` (~486 tok)
|
||||
- `4db3c1de6ac63ea0008c2a929d31af6a328b02f1f9ae4cca67aaf9e74d735e2b.json` (~2641 tok)
|
||||
- `4e1c9fda5ab7f3f66527cd52ef0fcc20daea535347855acfe4086cd0f9dd10c8.json` (~1221 tok)
|
||||
- `4e81d1fe6f1cfc63de7e1a59695f9687deecf540b45fa7df24141de82335a8b6.json` (~3036 tok)
|
||||
- `4e90715ba4f6253b1220ccc8949da57080ace7973196ecb257522a26b6288952.json` (~626 tok)
|
||||
- `4ee847c53c563a13117c4c6c9079895077ff7d18e951e8860530548e82f5f3e8.json` (~888 tok)
|
||||
- `4f1695f88fdc020f547c2cf509e08b8facfe69919b541b554c0aec76ec04537e.json` (~1432 tok)
|
||||
- `4f385d090c5ace09ef00e7883f4d965d6212218ea58df3f78088208fa47c7d56.json` (~8132 tok)
|
||||
- `4fbcc8c4f9a4190c3d5e6717b7d8315428abbdbc24b390b1081ad03954769178.json` (~181 tok)
|
||||
- `4fe83cc3758c5161a6c0c74e565d4e06a5eea66aa1488e2c1ac248a539742039.json` (~13032 tok)
|
||||
- `50b5f4dfe98634fea8d756c15c1b3f4d86fc1d607698133430210d0d97977856.json` (~1479 tok)
|
||||
- `51103bc49cc87487f9efa30dcffc5f7639f0cc6cbf8acc722f5da33fd0ce32e3.json` (~8308 tok)
|
||||
- `5175f1392d190fb958eb902f22aea50b3506648b739e0cb8a7bc42d389231e4b.json` (~4502 tok)
|
||||
- `51d646ce7e4c76a33d29d45d20149ff8825fe4339a842c2d000ed93d9e46f3de.json` (~12490 tok)
|
||||
- `527377e036844617174eac5b08965176bb127edc5db3d5a77f362b034a4ec4fd.json` (~10319 tok)
|
||||
- `5286722a3fefdea557784ae42c65304cf227e92c44178fff83b9f2394c57b2e6.json` (~5453 tok)
|
||||
- `528bfbd499082064cc9b015a4092ffba50f77e09f2f90b5497e20aec9aba24fb.json` (~138 tok)
|
||||
- `52a31389293007060719822853ae31ba48ea5a86df8441b9c191c64f173fdabf.json` (~2318 tok)
|
||||
- `52eabf2efd0f747f08eaf21143a5472f29e59591207fbc96262b97ff05925e6b.json` (~10316 tok)
|
||||
- `5343869ca34f5c819db0bccc7bc641c28bd99771118d48a495a4df736bc7cfd6.json` (~90431 tok)
|
||||
- `53630b8212623bc229ce74e756487117d202ec192b8120561d9fb5f19012b19c.json` (~1053 tok)
|
||||
- `53c97c6c796b513ab01170b56e6ace6f8725efa934165c6a84249fb0e278b687.json` (~4856 tok)
|
||||
- `53fb90c7ba54ed15ce868aec27ac9cf045a8b40311aa80dcd8c0c27619bb68c0.json` (~437 tok)
|
||||
- `546ac5f8e828ce834dbc0fd2e1af00b3970345404390f8e8de9d656977fb2761.json` (~2372 tok)
|
||||
- `547c8859d3083d2aa3df826d7d45b335b63abd89ecc10f67810628b642b26f28.json` (~32474 tok)
|
||||
- `54bf081b3ee033f99f334327084ec909aee9a0f93c01e9e7140a6da2e0e218b8.json` (~427 tok)
|
||||
- `556328f8d56c22698674e3a582461b4fd0b25468ffd0e633ab59b7441b9d1e52.json` (~1592 tok)
|
||||
- `5598204d609251195af1fd249f50dc13954de3767c7b1f6d3a5d7217f44ed180.json` (~1398 tok)
|
||||
- `55a60e8c9ee3b15597774bceaff343b2294c6bf4157ede4a0bc7e63338fb03a4.json` (~1627 tok)
|
||||
- `55a658f23a6ae2ae434c1847a8ce4e67d6077bb1d86cfdae1854e86b332c214a.json` (~3305 tok)
|
||||
- `55e566c3da1711786075f53bd5b5cf55c8c67385305c4e056e5c536471387fae.json` (~3852 tok)
|
||||
- `56138bbd8d51760f6d3569adae6d73f03a80a03921c90ad9428dfae1bc955894.json` (~59910 tok)
|
||||
- `577e37e81e94ff718f63cc8d59fb255f51ad4090898b88e7996ad50eeceb3379.json` (~4531 tok)
|
||||
- `57bfe3a23733334ec3ba0e0685f8c23e60f79d0c8a5db5f0e33dd1f050b747b0.json` (~13651 tok)
|
||||
- `57d9e640d91abd2f583860f3d63b54c0da0d89622194e945122d238869af3579.json` (~2262 tok)
|
||||
- `581c504c4d3fe6ca908dcfdb5bd9507b73f9b821e96fb15d58356ab6f2d04ccd.json` (~2270 tok)
|
||||
- `594aedf253eb6263edd24ffc2d34c93a770f4bdc0b728515eb053324773d4ddb.json` (~8323 tok)
|
||||
- `59f075695e0135dde52ba377bdd55298ecb1357cb177b5438d56d4244ec167f1.json` (~2354 tok)
|
||||
- `59f7956e28ec4fc2d018f85746017fdcfa3b6a124bbe9e6b2eb4cfc7e77b9be2.json` (~11412 tok)
|
||||
- `5a1508eee3b01cfd7586af0f88973dfc8c51ef9bdb8d3d6fcf7e40a61d663b0f.json` (~362 tok)
|
||||
- `5a1511c3eb48c941df71bc3f218854a6e5e8b3342f3a83d58b8bbc3adcd573dd.json` (~2093 tok)
|
||||
- `5a51f4134cc78b3e9b72124a8117a45e1d9a97d943172d29ba35c29fd844b12f.json` (~360 tok)
|
||||
- `5a92249e0d4f9dd34ff89497b4c540e6b84a672a455b0a5be5ca710b94e9eb79.json` (~4454 tok)
|
||||
- `5a974e6340597c0447c72b0d7bc977a668a8da2fac8c90c810e886c319e9bbd0.json` (~2118 tok)
|
||||
- `5a99edc88f1f75983b12b9b8931c7ba5717424024e2ffb000e19564f2956175c.json` (~2393 tok)
|
||||
- `5b5887beb614d310bb85633443a1e29df0dec82a39650daecc39d529ffe5ce37.json` (~4806 tok)
|
||||
- `5b710141d6eb0f199d33b695898e32519aca05ea857921dd62bf508c7f954948.json` (~567 tok)
|
||||
- `5b791c1c235d96fe2e512c3d6fedd8f44c5ee256db33beb485935a3f6cc3d1c1.json` (~6930 tok)
|
||||
- `5ba290aa8c7bdc0254d6b5efcaa55b3c85a821c51150bdebe182f29e37d9106b.json` (~14456 tok)
|
||||
- `5c1cd225c770bed4fc03e72d8433eb17a2193588c47d5abeb26580098aab6506.json` (~77 tok)
|
||||
- `5c3a1e3d92d85d4d34293b056e67c900f4050fc51af7aa5fa4d441070ecf536b.json` (~41692 tok)
|
||||
- `5cebb6a86cbc5a90d803973f1365c1673f3a11bd36cfc5178bdc985a03950378.json` (~8230 tok)
|
||||
- `5d7b2f244a9e7100f16139db782e80265d4724d5cb53b7c51fab1aeaaf1fb7b6.json` (~1313 tok)
|
||||
- `5d996b732909e4428bb55522f4fdad491a98075c3aa256d7a816956bd42adac7.json` (~984 tok)
|
||||
- `5da97b26e48c2ea5c6df990bedce598cfd8052022102fa7a7e4c499c6f1374f5.json` (~1773 tok)
|
||||
- `5eb698f53d93049c137820a52db3d1a7c451be085cea9d956a06b44671034787.json` (~1504 tok)
|
||||
- `5ed29f8b6584e6bb033cd37fb6ccdf486254adda489f989641a3dfdf1c595cd4.json` (~726 tok)
|
||||
- `5ede9a0001022535c762bd22ae452960e9664ea3487e3bcd04a9cc3368b9b819.json` (~2470 tok)
|
||||
- `5f1dd49f629f4e9163fb4f2dc4ba3c810a01e6c6604835843560f1fab6303bd6.json` (~506 tok)
|
||||
- `5f4bcb8abfd96052ce4500607dd3d9426acb458a5cba0a7a57f641f6b7d5a95f.json` (~801 tok)
|
||||
- `5f522fbbd977c3b08acea207a56d99aed5069c1b882a186498c8f0cd0957493c.json` (~2972 tok)
|
||||
- `6041d9e4749c93c1268d8ddce42d2eed66ab5b1dd739ef9fd28bee04f745bb48.json` (~3610 tok)
|
||||
- `6101027f230f560481292b1839c195d6a761a1a40d615f2aa6c7ab8dcb0db512.json` (~20625 tok)
|
||||
- `6138099a827202f5f984212ac5a92cdd0181489a14a0f341b27169d40695fa9b.json` (~12476 tok)
|
||||
- `616ea854efce5a7e95c1739d6400bb48fd63c7cee591a710c66a2bcc9e70fd20.json` (~4511 tok)
|
||||
- `6180aec3d86b1cb2980831e6c75a8ac6216fd14493f52a1539ad73a750bd10b7.json` (~5053 tok)
|
||||
- `61f1333461274a206f965221fa31fd7ce681795ce2c4271c9455759288e83655.json` (~4360 tok)
|
||||
|
||||
## src/controller/
|
||||
|
||||
- `join.c` — include "config.h" (~2783 tok)
|
||||
- `meson.build` (~1216 tok)
|
||||
|
||||
## src/controller/testsuite/
|
||||
|
||||
- `controller_test.c` — include <argp.h> (~1487 tok)
|
||||
- `test_join_redirect.c` — include <stdint.h> (~937 tok)
|
||||
|
||||
## src/include/
|
||||
|
||||
- `client.h` — if !defined(_CLIENT_H_) (~1164 tok)
|
||||
- `mylock.h` — if !defined(_MYLOCK_H_) (~273 tok)
|
||||
- `xcectx.h` — if !defined(_XCECTX_H_) (~678 tok)
|
||||
|
||||
## src/include/xce/
|
||||
|
||||
- `client_failover.h` — ifndef XCE_CLIENT_FAILOVER_H (~618 tok)
|
||||
|
||||
## src/lib/
|
||||
|
||||
- `client_failover.c` — define _GNU_SOURCE (~4435 tok)
|
||||
- `client_internal.h` — if !defined(_CLIENT_INTERNAL_H_) (~3500 tok)
|
||||
- `client.c` — include <errno.h> (~4370 tok)
|
||||
- `meson.build` (~1535 tok)
|
||||
- `xcectx.c` — include <errno.h> (~2223 tok)
|
||||
|
||||
## src/lib/testsuite/
|
||||
|
||||
- `failover_harness.h` — Declares xce_cluster_config_t (~1393 tok)
|
||||
- `failover_harness.h` — test-only header for the libxce-failover harness: fo_master_cursor_t, FO_STATUS_TABLE rows, fo_run_threads(); no production failover symbols (~900 tok)
|
||||
- `test_libxce_retry.c` — Declares struct (~18140 tok)
|
||||
|
|
|
|||
1934
.wolf/buglog.json
1934
.wolf/buglog.json
File diff suppressed because it is too large
Load diff
|
|
@ -9,7 +9,18 @@
|
|||
<!-- How the user likes things done. Code style, tools, patterns, communication. -->
|
||||
|
||||
## Key Learnings
|
||||
- (2026-06-20, waldump tooling) The `xce_test` binary (`src/controller/meson.build`) builds from `testsuite/test.c` ONLY + `libxce_dep` — it does NOT link `persist.c`/`job.c` (those are in the `controller_unit_test` target). The investigation doc's claim that xce_test "links persist.c/job.c" was WRONG. BUT capnp `read_Job` is generated into `controller.capnp.c` which IS compiled into `libxce`, so it's reachable transitively. To USE capnp/zlib symbols from test.c you must add `c_capnproto_dep` + `zlib_dep` to `xce_test_deps` (static-lib deps don't auto-propagate through `declare_dependency(link_with: libxce)`). For a zero-alloc jobid decode read `struct Job.jobid` directly after `read_Job` — skip the allocating `decode_Job`. WAL CRC reader template = `persist_test_durable_lsn_on_disk` (persist.c ~888): seg files `seg-%016llx.wal` under `<node>/data/wal`, 40B `wal_entry_hdr_t`, crc32 over `WAL_HDR_CRC_SIZE`(=32) header bytes + payload, prev_crc chain, all-zero hdr = clean EOF (pre-alloc). Offline `waldump` lives in `testsuite/waldump.{c,h}` + `cmd_waldump` in test.c top_commands.
|
||||
- (2026-06-18, ha-commit-durability task 5.2 VALIDATION) Deterministic E2E on the FIXED build = GREEN, stable 3/3 runs (9/9 probes PASS), in-window primary fdatasync=2 on EVERY probe (vs 0/RED on the unfixed 1.3 build) — the RED→GREEN flip is the deterministic acceptance and is confirmed. Run via `nix develop ./src --command bash -c 'bash src/tools/test-commit-durability.sh --deterministic <deploy>'` background+poll worktree log; cold election ~15s, 3 probes, exit 0. The ROLLING SIGKILL SOAK is host-verification-needed in-sandbox: rounds 1-2 PASS (jobs survive failover), then round 3 onward "no new primary elected within 40s" and every "DATA LOSS" report is a query against a cluster with NO live primary (same 5 jobs frozen, never re-acked). ROOT CAUSE of the collapse is the durability fix's OWN promotion fence working AS DESIGNED: ctrl-1 won PreVote for term=5, computed safe_committed_lsn=13, then `persist_recover_fatal_if_unrecoverable` FATAL-terminated ("recovery-truncated log ... no healthy peer reachable ... durable_floor_lsn=13 < hwm=13 ... terminating rather than serving divergent state, Req 2.4") — refusing to let a structurally-truncated node lead and overwrite the good timeline (exactly the bug-661 failure mode the fix prevents). Under aggressive rolling-kill+immediate-restart the fence repeatedly self-terminates recovering nodes → cluster drops below quorum → liveness collapse, NOT durable loss. PROOF it's not loss: committed entries still on disk after the run (ctrl-3 WAL has seg-1 + seg-15; ctrl-2 seg-1) and 0 live controllers at end. So the soak's exit=1 / "DATA LOSS" verdict is a HARNESS ARTIFACT (it equates empty-joblist-with-no-primary to loss) — the soak harness conflates election-stall liveness failure with durability loss. The deterministic flip is the must-have and passed; the live multi-round soak needs a real host (slower election timing, no sandbox reaping) to distinguish fence-driven safety self-termination from a true regression. Did NOT patch production (5.2 is validation-only).
|
||||
- (2026-06-18, ha-commit-durability task 5.1) End-to-end durable-majority commit verified via 3 integration tests in test_commit_durability.c (NO production wiring gap found — the 2.x/4.x wiring already connects append→flusher fdatasync→durable_lsn→durable-advance wake→majority_candidate_locked self-cap→committed_lsn→commit barrier). (1) test_durable_majority_commit_end_to_end: REAL persist + REAL flusher + commit_engine(n_peers=3) + 2 registered standbys; with persist_test_set_force_fsync_fail(1) a FULL standby-ACK majority does NOT advance committed_lsn (durable precondition gate, Req 1.1/1.4) and a WAIT submit on its own thread does NOT release; after clearing the fault + persist_flush, durable_lsn advances, the durable-advance cb re-derives majority and the WAIT releases rc=0. GOTCHA: persist_wal_async fans out via repl_broadcast to registered standby eps — dummy int eps cause an ASAN global-buffer-overflow in dataq_enq; FIX = install the XCE_TESTING `repl_broadcast_send_hook` (drop hook, same as test_commit_engine.c) before appending, NULL it after. (2) test_sole_durable_contributor_loss_survives: models the N=3 durable-majority property (committed ⇒ ≥2 durable copies) — write the same op durably to two independent persist dirs (one via persist_wal_async+flush, one via persist_wal_write_replicated which fdatasyncs inline), rm -rf one, prove the survivor still recovers it CLEAN; live multi-host failover proof deferred to 5.2 (Req 1.5/2.5). (3) test_durability_work_off_iomgr_preserves_liveness (Req 1.6): new XCE_TESTING seam persist_test_last_fdatasync_thread() records pthread_self() at the successful flusher fdatasync site (flusher_flush_once, under p->lock); test asserts it != caller thread, and that heartbeat-style atomic reads on the caller thread complete <50ms while flushes are in flight. RED PROVEN by injecting the pre-2.2 bug (cap=highest_appended_lsn): the commit.c:162 `assert(cand <= persist_durable_lsn)` aborts when the ACK-majority tries to advance committed over a non-durable entry. 4.3's bounded-window catch-up TIMER left as 5.3 follow-up (immediate fail-loud is safety-conservative). Suite Fail:0 ASAN x2; seam nm-excluded from xce_controller. 34 durability tests (was 31).
|
||||
- (2026-06-18, ha-commit-durability task 4.3) Pre-promotion eligibility gate + fence release. Added TWO functions in ha.c (decls in ha.h): `ha_promotion_gate_decision(bool truncated, uint64_t durable_floor, uint64_t safe_hwm, bool peer_reachable, bool catch_up_succeeded) -> recover_decision_t` (PURE — builds recover_policy_inputs_t with `durable_below_hwm = durable_floor < safe_hwm` and delegates to `persist_recover_decision`, so the 3.2/3.3 decision table is the SINGLE source of truth, DRY) and `ha_evaluate_promotion_fence(ha, ...)` (LIVE — ELIGIBLE clears `ha->in_catch_up`, FENCE sets it, FATAL routes through `persist_recover_fatal_if_unrecoverable`). WIRED into `raft_cb_become_leader` (ha.c ~703) AFTER `persist_truncate_wal_above(safe_lsn)` (so `persist_durable_floor` reflects the 2.2 clamp) and BEFORE the CTRL_PRIMARY store; on `!= ELIGIBLE` it logs WARN and `return`s WITHOUT becoming primary (so a test-hooked FATAL never falls through to leadership). `safe_lsn` lifted to fn-scope var `promotion_safe_lsn` (the cluster HWM from voter replies — `ha_compute_safe_committed_lsn` arithmetic UNTOUCHED). peer_reachable derived from surviving granted vote eps (`ectx->votes[i].vote_granted && .ep != NULL`). GATE SKIPPED when `c==NULL` (the `raft_cb_become_leader_for_test` shim has no persist) — that is WHY existing election/role/convergence suites (all driven via the c==NULL shim or `ha_promote_to_primary`, a SEPARATE function with no gate) pass unchanged with ZERO fixture edits. Turned 3.2 rows 1/2/3/5/6 GREEN (0 Ignored) by driving the live gate (rows include a fence-set then catch-up→release sequence proving Req 2.6); added `test_promotion_gate_hollow_refused_at_minimal_quorum` (Req 2.5). test_commit_durability.c needed `#include "ha.h"` (ha.h forward-declares cluster_t, self-contained). LIVE-TIMER FATAL DEFERRED to 5.1 (allowed): the gate evaluates `catch_up_succeeded=false` at the synchronous promotion instant, so a truncated/behind node with NO peer FATALs immediately (correct terminal) — the bounded-window TIMER that gives DEFAULT_RECOVERY_CATCHUP_WINDOW_MS to find a peer before FATAL is the live-timer aspect 5.1 owns; the decision + fence + release are all live HERE. Suite: 31/0 durability (0 Ignored), full controller direct-binary = only the pre-existing barrier baseline fail (overlay-/tmp EIO), meson controller test OK Fail:0.
|
||||
- (2026-06-18, ha-commit-durability task 3.3) Boot-policy FATAL on bounded-unrecoverable recovery. SPLIT into a PURE decision + a SIDE-EFFECTING terminate so the decision is unit-testable without aborting: `recover_decision_t {ELIGIBLE=0,FENCE=1,FATAL=2}` + `recover_policy_inputs_t {verdict, durable_below_hwm, healthy_peer_reachable, catch_up_succeeded, durable_floor_lsn, cluster_hwm_lsn}` in persist.h; `persist_recover_decision(const inputs*)` (pure, no side effects) + `persist_recover_fatal_if_unrecoverable(const inputs*)` (logs truncation boundary at ERROR then terminates) in persist.c. FATAL ONLY when (truncated OR below-HWM) AND no healthy peer AND !catch_up_succeeded (rows 4,4b). A reachable healthy peer is NEVER FATAL (rows 3,5 → FENCE) — this is the invariant 3.3 owns; the full FENCE→ELIGIBLE eligibility is 4.3. TERMINATE SEAM: added `persist_fatal_hook` (extern void(*)(const char*), declared in persist.h, DEFINED =NULL in persist.c — NOT under XCE_TESTING so production links it as a NULL global → production path is `tlog_exit(); abort()`). Mirrors the existing `ha_fatal_hook` convention (ha_convergence.c:58 / ha.h:1045). Tests install the hook to capture FATAL without dying (test_commit_durability.c dur_fatal_hook + g_fatal_hook_calls counter). Bounded window = `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS 30000` (persist.h) — FINITE, never indefinite. Production nm-verified: persist_recover_decision/persist_recover_fatal_if_unrecoverable are real T symbols, XCE_TESTING seams excluded. Rows 4/4b + a focused peer-never-fatal test PASS; rows 1/2/3/5/6 stay TEST_IGNORE for 4.3 (suite GREEN). `tlog_exit()` is reachable in persist.c transitively (no extra include needed — ERROR/WARN already pull tlog via xcelog.h chain).
|
||||
- (2026-06-18, ha-commit-durability task 3.1) Recovery-integrity verdict: `recover_verdict_t {RECOVER_CLEAN=0, RECOVER_TRUNCATED=1}` in persist.h (enum declared BEFORE struct persist_s since the struct field uses it); `recovery_verdict` field on persist_t (NOT ha_state_t — keeps the boundary in persistence, avoids pulling in ha.c); `replay_wal` gained a `recover_verdict_t* verdict_out` out-param (single caller: persist_recover at persist.c). Classification: clean stops = `hn==0` EOF, all-zero pre-alloc sentinel, end-of-segments → CLEAN; structural corruption = CRC chain break (persist.c ~1463/1471), CRC mismatch (~1500), short payload read (~1490), AND `0<hn<sizeof(hdr)` partial-header / `hn<0` read-error → TRUNCATED. Verdict init CLEAN once at fn entry, only ever SET to TRUNCATED (a later clean segment never resets it). Snapshot-sentinel + clean replay is CLEAN (load_snapshot sets checkpoint_lsn from the 16-byte sentinel; replay only adds entries ABOVE it, no sub-sentinel entries = normal). 3.1 ONLY classifies + sets the flag (accessors persist_recovery_verdict / persist_recovery_truncated) — does NOT FATAL (task 3.3) and does NOT fence promotion (task 4.3 consumes it). Verdict is about LOCAL log structure, NOT being behind the cluster (a clean log ending below high-water is CLEAN; freshness/promotion fence catches "behind"). Test crafts corruption by append+flush then pwrite-flipping the on-disk first entry's `crc` field; WAL seg path is `<tmpdir>/<name>/data/wal/seg-%016llx.wal` (name="ctrl-test" in make_durability_persist — first entry is LSN 1 ⇒ seg-...0001.wal, LSN 0 reserved).
|
||||
- (2026-06-18, controller test ordering) PRE-EXISTING FAILURE (NOT a regression): `test_fact_hook_append_is_retained_and_nonblocking` (test_commit_barrier_rollout.c:446 "a covering ACK must advance committed_lsn over the retained fact") FAILS even on a clean baseline (git-stash-confirmed) and FAILS in isolation (`controller_unit_test -t test_commit_barrier_rollout`) — a timing-sensitive commit-barrier test, unrelated to recovery/persist changes. Also: leftover `/tmp/xce_*` dirs cause a cascade (errno=5 EIO fdatasync noise + extra spurious FAILs like test_demotion_replay_yields_committed_only); `rm -rf /tmp/xce_*` before each suite run clears it and the run is stable at exactly that one pre-existing fail. When validating a persistence task, confirm a failing barrier test is the baseline one before treating it as your regression.
|
||||
- (2026-06-18, ha-commit-durability task 1.1) Controller CLI flags follow a fixed 6-point argp pattern in `src/controller/main.c`: (1) `#define OPT_*` key above ASCII, (2) field in `struct arguments`, (3) entry in `options[]`, (4) `case` in `parse_opt` (validate, e.g. `atoi`<0 → `argp_error`), (5) default-zero init in `main()` (0 = "not passed"), (6) assign into `myconf` (struct in `src/include/myconf.h`, NOT `src/controller/`). `persist_config_t` overrides live in `src/include/persist.h` as `static inline` helpers next to `persist_config_defaults()`; there are TWO `persist_config_defaults()` callers to keep in sync — `cluster.c` phase2 (~L1267) and `myapp.c` step 6 (~L164). GOTCHA: capnp generates a same-named `PersistConfig_set_walSyncMs` (wire serialization) — the compiler's "did you mean" suggests it; it is NOT the runtime override helper. Test home: `testsuite/test_cluster_persist.c` (already covers `persist_config_defaults` + `wal_sync_ms`).
|
||||
- (2026-06-17, bug-660 slow-election convergence) `ha->last_activity` is DUAL-PURPOSE and that conflation is a trap: it's the watchdog grace clock (reset to now on every `ha_arm_heartbeat_timeout_vtask`, incl. `ha_prevote_rearm_liveness` after each lost PreVote) AND the source for `ha_heartbeat_is_fresh` (the `reason=fresh_heartbeat` PreVote/vote deny). So a node with NO live primary keeps refreshing its own "fresh heartbeat" every failover cycle and denies every legitimate higher-term candidate → elections take 11+ rounds / minutes to converge (a standby denied for 2.5 min after its primary was killed). FIX: the eligibility gate must also require a LIVE primary connection. `raft_cb_vote_eligible` (ha.c:311) checked `is_fresh` ALONE; `handle_request_vote` (ha.c:2205) already required `primary_ctrl_fd >= 0 && is_fresh`. Adding `ha->primary_ctrl_fd >= 0 &&` to ha.c:311 (+ cluster.c:526 reason logging) makes a dead/absent primary (fd=-1 on init / RDHUP / watchdog-fire; real fd only set at config_repl.c:1451 on first ctrlhello) GRANT → converge in ~1 round. The existing fresh-deny tests already set `primary_ctrl_fd=999` and documented "the deny gate requires a live connection AND a fresh heartbeat" — production just didn't enforce it. Diagnostic signature: `ag "PreVote DENIED" logs/` shows `reason=fresh_heartbeat` long after the primary's last log line. Test: test_pre_vote_dead_primary_grants_despite_fresh_activity (controller_unit_test -t test_ha_vote_gate).
|
||||
- (2026-06-15, ha-rejoin bug-340) A killed+restarted controller loops elections forever because the PRIMARY never evicts the dead/wedged outbound standby endpoint. primary_heartbeat_fn (ha.c) has two steps: (1) NOP each ep in ha->standby_endpoints, (2) reconnect to any peer NOT present in standby_endpoints. A wedged ep (peer died → handle_error did EPOLL_CTL_DEL but left it in the array; epoll MOD then returns ENOENT → the `WARN endpoint_update_epoll rc1=-1` spam) makes step 2 think the peer is still connected → reconnect SUPPRESSED → restarted node never gets a ctrlhello → re-elects every heartbeat_timeout (term climbs unbounded). xce_nop_req returns >=0 on enqueue even when the epoll re-arm fails, so the heartbeat layer can't see the wedge. FIX: evict in step 1 using endpoint_t.ts_last_in (existing wall-clock field, stamped by handle_input) — if ts_last_in>0 && now-ts_last_in>heartbeat_timeout, zarray_del from standby_endpoints + repl_remove_standby; reconnect then re-establishes the channel and ctrl-1 settles via ha_discover_primary (NO term gate — accepts any primary msg while ELECTING regardless of its own high term). Gotchas: (a) ha_arm_heartbeat_timeout_vtask ALREADY resets last_activity=now, so 'become_follower doesn't reset liveness' is NOT a bug — the ~timeout loop period proves the watchdog is correct; the sole cause is the broken channel. (b) standby_endpoints holds TWO ep types (promoted vote_ep_ctx_t with ctx!=NULL, and heartbeat standby_ep_ctx_t with ctx==NULL) sharing first 4 fields; don't free vote-ep data (UAF — live vctx pointers). (c) Run one suite: controller_unit_test -t test_ha_liveness, ASAN_OPTIONS=detect_leaks=0.
|
||||
- (2026-06-17, bug-646) The `WARN endpoint_update_epoll rc1=-1` spam (same one called out in the bug-340 learning above) was a SILENT-SWALLOW: `endpoint_update_epoll` (netio.c) logged the `iomgr_update_poll` failure but returned 0, leaving the poll interest unarmed. `iomgr_update_poll` uses EPOLL_CTL_MOD → ENOENT when the fd isn't registered (e.g. after a failed reconnect, or a wedged ep that handle_error EPOLL_CTL_DEL'd but left in the array). Consequence on the CLIENT: a `jobsub` after failover hangs forever in `cbuf_get` (xceapi.c, no timeout) while `io_worker` sleeps in `epoll_wait` — because the request bytes were never flushed (EPOLLOUT never armed) and no socket-drop fired `handle_disconnect`. FIXED: endpoint_update_epoll now returns -errno; `endpoint_enq` runs `handle_error(ep)` on arm-failure (wakes the waiter via handle_disconnect, failover rotates); `handle_output` propagates too. Non-obvious facts that ruled out other causes: (1) server commit barrier is bounded — DEFAULT_SYNC_TIMEOUT_MS=500 (config.h:93); (2) `ha_dispatch_gate` (ha.c:1628, called at cluster.c:251) ALREADY replies E_XCE_STANDBY / -XCE_ERR_NOT_READY for writes to a non-primary / quorum-less-primary — so "writes aren't role-gated" is NOT a gap; (3) client connection-break IS handled (handle_disconnect injects -ECONNRESET into wbuf). Test: test_netio_arm_failure.c.
|
||||
- (2026-06-17, bug-648 + VERIFIED SSL close semantics) OpenSSL 3.x peer-close behavior, confirmed against official docs (SSL_get_error/SSL_read man pages) AND a live probe on this build (OpenSSL 3.6.0): a peer close makes **SSL_read() return 0** (never <0 for a clean FIN, never WANT_READ) and the meaning is carried by **SSL_get_error(), NOT the return value or errno**. Mapping: clean close_notify → SSL_ERROR_ZERO_RETURN (errno 0); **unexpected EOF (FIN, no close_notify) → SSL_ERROR_SSL since OpenSSL 3.0** (pre-3.0 it was SSL_ERROR_SYSCALL + errno 0 — documented in the SSL_get_error NOTES); RST → SSL_ERROR_SYSCALL errno=ECONNRESET(104). **errno is authoritative ONLY for SSL_ERROR_SYSCALL**; for SSL_ERROR_SSL the error QUEUE is authoritative and any errno is incidental — we measured EPIPE(32), which is the leaked side effect of OpenSSL's internal alert-WRITE (raw write() on a broken conn → EPIPE; raw read() → EOF/0 or ECONNRESET, never EPIPE). Consequence: the old `rc = -errno` in io_fsm_do swallowed every documented EOF case (errno 0 → 0). FIX: io_classify_io_error() (pure, non-static, unit-tested in test_netio_close_propagation.c) returns 0 only for WANT_READ/WANT_WRITE + EAGAIN syscall, else negative ((ecode!=0)?-ecode:-ECONNRESET). DESIGN CONCLUSION (user, reaffirmed [[feedback-fail-loud-over-timeout]]): **no client reply-wait timeout** — close detection is made deterministic at the source instead. CAVEAT: on THIS 3.6 build a process-death FIN gave SSL_ERROR_SSL/errno=32 which the OLD code already propagated, so bug-648 is likely NOT the sole cause of the controller-kill permanent hang — that still implicates the reconnect/SSL-handshake-stall path (client TCP-connected, state RUN, handshake never completes → request queued, never flushed, io_worker idle in epoll_wait). Verify live before closing that out.
|
||||
- (2026-06-15, ha-election bug-339) Candidate-advertised RequestVote LSN MUST equal the voter-side freshness floor. Voter floor = ha_vote_self_freshness(ha, config_tag, applied_lsn) = max(applied_lsn, last_known_committed_lsn) (cluster.c OPCode_requestVote). On a STANDBY, persist.committed_lsn NEVER advances (only the primary commit engine writes it at persist.c:1192); standbys advance applied_lsn (ha_apply_wal_catchup_entry) + last_known_committed_lsn (heartbeat replies). So a candidate that advertises persist.committed_lsn under-advertises vs its own floor → after a primary dies, every surviving standby advertises stale-low LSN while each peer's floor is high → freshness gate (cand_lsn<my_lsn) denies on BOTH sides → no quorum → infinite RDHUP→ELECTING→STANDBY loop, terms climb in lockstep, nobody ever become_leader. Boot path (myapp.c:217) already uses applied_lsn; only the RDHUP failover path (ha_on_primary_ctrl_fd_rdhup) was inconsistent. Fix: compute cand_tag/cand_lsn via ha_vote_self_freshness once and feed BOTH the elect_ctx (votes[0]+self_*) AND the ha_rdhup_election_hook args. Symmetric advertised==floor guarantees the freshest node is granted by all peers. Diagnostic signature in logs: both nodes only raft_cb_become_follower (never become_leader) + RDHUP every ~heartbeat_timeout.
|
||||
- (2026-06-14, election-safety) ha_on_primary_ctrl_fd_rdhup MUST mirror the ha_start_election_async no-config voter-only guard: a config-less node (myconf->config_version_tag==0) with candidates_count>1 must NOT start an RDHUP re-election (it would tie-grant with another tag=0 node — freshness gate only denies a LOWER tag — and elect a config-less primary that pushes an empty config). Guard sits AFTER the WAL-truncate block, BEFORE elect_ctx alloc/hook/raft_on_leader_lost; sets role=CTRL_STANDBY and returns. RDHUP integration/failover tests that assert election-after-RDHUP use make_cluster/memset which leaves config_version_tag=0 + candidates_count=2 — they were incidentally testing the buggy condition; give them a nonzero tag (0x1234) so they exercise the legitimate CONFIGURED-node path. RDHUP handlers called with c==NULL early-return before the guard (unit-mode tests assert only the CTRL_ELECTING transition). CTRL_PRIMARY enum==0, CTRL_STANDBY==1, CTRL_ELECTING==3 (test printf "role=0==CTRL_PRIMARY" is expected). Full controller suite: meson driver SIGTERMs at 300s — run ./xce.build/controller/controller_unit_test directly with ASAN_OPTIONS=detect_leaks=0.
|
||||
- (2026-06-13, barrier-rollout 5.2) push_config_to_standbys now RETURNS the configAck barrier result (0/-ETIMEDOUT/-EINVAL), push count via acked_out; the configAck branch of repl_wait_majority_ack reads entry_lsn as the push tag and r->ce->n_peers (set-once, lock-free) as the denominator, ce==NULL -> -EINVAL fail-loud. ha.c/join.c streaming pushes discard the return BY DESIGN (log-only); only do_upgrade_config_with_bytes treats it as authoritative.
|
||||
|
|
@ -69,9 +80,13 @@
|
|||
|
||||
## Do-Not-Repeat
|
||||
|
||||
- (2026-06-21, bug-850) Do NOT fix "a rejoining standby races live replication ahead of catch-up" by defaulting `repl_add_standby` to `in_catch_up=1`. It has a huge blast radius: a fresh standby then does NOT count toward `repl_active_standby_count` write quorum until it converges, so EVERY commit-WAIT path stalls — the full controller suite TIMED OUT at 300s (commit.c "WAIT deadline expired ... committed_lsn=0"), because unit tests register standbys + commit directly without running the heartbeat dispatch that would clear the flag. The correct, surgical fix is a CONTIGUITY GUARD at the persist layer (Raft log-matching): `persist_wal_write_replicated` rejects a forward gap (`lsn > current_lsn`) with `-EAGAIN` (no write, no current_lsn jump); the apply path treats `-EAGAIN` as a quiet skip (no ack, no ERROR spam) so the ACK-driven catch-up continuation re-streams from `current_lsn` and the log fills in order. Zero quorum-semantics change → zero suite regression (129/0/0). Relies on bug-848 (advertise `durable_floor == current_lsn-1`) so catch-up always starts AT `current_lsn` and is never spuriously rejected.
|
||||
- (2026-06-18) When a task changes commit semantics to be ASYNC-durable (self caps at durable_lsn, advanced by the demand-driven group-commit flusher), every test that SYNCHRONOUSLY asserts "covering ACK ⇒ committed_lsn advances" must first force durability (persist_flush) or bounded-poll. ha-commit-durability 2.2 primed durable_lsn in the in-memory barrier_setup_vcpus stubs but MISSED the real-persist test_fact_hook_append_is_retained_and_nonblocking → 6/6 deterministic FAIL. Do NOT trust an impl-note that blames a test failure on a /tmp-overlay fdatasync-EIO without re-verifying: fdatasync returned rc=0 on /tmp here and the test passed pre-spec — it was a real durable-gating regression. Also: meson 'Fail:0' masks Unity sub-group failures — always gate on ANSI-stripped ':FAIL:' count from the test binary's stdout, not the meson summary.
|
||||
|
||||
<!-- Mistakes made and corrected. Each entry prevents the same mistake recurring. -->
|
||||
<!-- Format: [YYYY-MM-DD] Description of what went wrong and what to do instead. -->
|
||||
|
||||
- [2026-06-17] Do NOT invoke handle_error()/an endpoint err_handler from the SYNCHRONOUS send path (endpoint_enq / request.c senders). The err_handler is an io_worker-thread callback that may take ha->lock (vote_ep_err_handler, ha.c:3478) or ctx-level locks; the send path runs on the caller's thread, which often already holds that lock (controller primary_heartbeat_fn holds ha->lock while sending heartbeat NOPs) → self-deadlock on a non-recursive mutex (bug-652, a regression from bug-646). Correct pattern: endpoint_enq RETURNS a retriable error (-ECONNRESET) and the caller propagates it (request.c senders return it — bug-650); the real teardown stays on the io_worker thread (io_handler→handle_error), which holds no caller locks. General rule: a libxce send must never run endpoint teardown/callbacks inline — only surface the error.
|
||||
- [2026-06-09] Do NOT make the duplicate-vote deny candidate-agnostic ("term-only deny"). Raft §5.2 grants iff votedFor ∈ {Nil, candidateId} — same-candidate re-grant must stay idempotent (network retransmission path). An uncommitted ha.c change did this and broke test_vote_granted_to_same_candidate_same_term; ha.c must match raft.c's Step 4 semantics (deny only a DIFFERENT candidate in the same term). See bug-338.
|
||||
- [2026-06-09] The "self" voted_for_name placeholder only exists in test mode; production raft_cb_persist records the node's REAL name for a self-vote. Any check for "did I vote for myself" must compare against BOTH "self" and c->myconf->name (see myapp.c leader_known).
|
||||
- [2026-06-15] SOLVED (bug-563): "jobsub ok but joblist/workerlist empty on the HA primary; reslist works". ROOT CAUSE: `cluster_start()` is the ONLY setter of `c->running = 1` and it is ORPHANED — zero callers (grep `cluster_start(`). The HA `myapp_run` path starts iomgr/vtaskmgr/delaymgr/persist itself but never calls `cluster_start`, so `c->running` stays 0 on the primary. `check_controller()` (the streaming gate `xce_reply_get` uses for joblist/workerlist) returns `-EINVAL` when `!c->running`; `xce_reply_get` then sets the chunk count `nc = 0`, shipping an EMPTY list even though `collect_jobs()` found the rows. `reslist` passes `checkf=NULL` so it bypasses the gate — that asymmetry is the tell. Single-node "works" because the legacy `cluster_init` path called `cluster_start`. FIX: `c.running = 1;` in `myapp_run()` after `persist_start()` (src/controller/myapp.c). RULE: when only SOME list ops are empty but the data is present server-side, check the `xce_reply_get` `check_*` predicate — a `<0` return silently zeroes the chunk (`nc=0`), not an error. To prove it: log `nc`/`checkstatus` in the `xce_reply_get` send loop. Reproduce HA bugs locally with `/xce-build` + a 3-node deploy-ha under ASAN rather than theorizing. (My earlier UAF/stale-binary guess here was WRONG — disproven by "crash gone, binary current".) Still-latent: `cluster_start` also armed `check_workers_timeout` (delaymgr) — that side-effect remains unrestored on the HA path.
|
||||
|
|
@ -109,6 +124,7 @@
|
|||
- (2026-06-11) Scheduler-bound job egress has TWO seams, not one: full push `xcesched_push_jobs` (sched.c:169, on scheduler join) AND incremental JOB_NEW push `sched_job_new_fun`→`xcesched_push_joblist` (sched.c:236-248). Status reads (apiserver.c:83, opget.c:173) share `collect_jobs` (dataex.c:294) and must stay unfiltered per umbrella deviation.
|
||||
- (2026-06-11) No saved-argv/execv machinery exists anywhere in src/controller or src/lib — demotion self-restart is net-new; `job_t` is a thin wrapper `{ xce_job_t job; }` at src/include/xce/types.h:483-485 (controller-local fields go on the wrapper, wire type untouched).
|
||||
- (2026-06-16, ha-prevote cleanup) The HA election carrier `ha_elect_ctx_t` (`ha->elect_ctx`) is deliberately leaked one-per-round on BOTH the real-election and PreVote paths — vote endpoints hold a bare `vctx->ctx` pointer freed async by endpoint_fini, so freeing the carrier risks UAF; there is NO real `ha_election_cleanup()`. Pre-existing + bounded (rare failover events). Deferred by user decision to follow-up spec `.kiro/specs/ha-elect-ctx-lifecycle/` (refcount-the-carrier or teardown-ordered free, both paths). Do NOT attempt a drive-by free — UAF on failover is worse than the bounded leak. The ha-prevote 3.1 log-reason nit is RESOLVED (commit 47f7338): `ha_heartbeat_is_fresh` un-static'd + declared in ha.h; cluster.c denial-reason now mirrors `raft_cb_vote_eligible` order exactly.
|
||||
- (2026-06-17, bug-646 client-hang fix) User rejected adding a client reply-wait timeout (`cbuf_timedget`) as a fix for the failover hang: "a timeout may hide potential issues and make things more complex." Principle adopted: **fail loud at the source, never mask with a backstop timeout.** The hang was a netio silent-swallow (`endpoint_update_epoll` returned 0 on `iomgr_update_poll`/EPOLL_CTL_MOD failure), so the fix surfaces the error and runs the error path (wake the waiter via handle_disconnect → failover rotates), rather than papering over the missing reply with a deadline. When choosing between "add a timeout" and "propagate the real error," prefer propagation on this codebase.
|
||||
- (2026-06-16, ha-prevote impl) The controller `"controller test"` 300s TIMEOUT under `meson test` is a WALL-CLOCK WRAPPER ARTIFACT, not a hang/failure: running the binary directly (`ASAN_OPTIONS=detect_leaks=0 ./xce.build/controller/controller_unit_test`) completes all 564 tests incl. the slow `test_commit_barrier_reply` (61 tests, real 500ms commit barriers) with 0 failures. For a true controller pass/fail, run the binary directly (no cap) or filter with `-t <suite>`; do NOT treat the meson timeout as a regression. ha-prevote feature COMPLETE: 11 tasks, raft suite 123/0, controller 564/0, real-vote path byte-for-byte unchanged (0 raft.c deletions), commits 0d66026..e6b234a.
|
||||
- (2026-06-15, ha-prevote design review) PRODUCTION vote-reply tally path = `raft_on_vote_reply` in the raft library, NOT `ha_elect_ctx_t`. `vote_reply_dispatch` (ha.c ~2630-2650) calls `raft_on_vote_reply` directly in production; the `ha_elect_ctx_t` / `handle_request_vote_reply` / `ha_elect_evaluate_and_signal` path runs ONLY behind `#ifdef XCE_TESTING` via `xce_vote_reply_hook` (plus the startup/first-boot `ha_elect_ctx_t.done` semaphore). The raft library owns quorum tracking and fires `on_become_leader`/`on_become_follower`. Do-Not-Repeat: when mirroring the election reply path, mirror `raft_on_vote_reply` (raft lib), not `ha_elect_ctx_t`. (An earlier exploration mislabeled ha_elect_ctx_t as the operative tally; caught in design validation.)
|
||||
- (2026-06-15, user decision, ha-prevote RN-2) Leader-stickiness window for PreVote = reuse `ha_heartbeat_is_fresh()` / `heartbeat_timeout_ms` (10s), gated on role==CTRL_STANDBY. Rationale: the stickiness window MUST equal XCE's failure-detection window or you get either a disruption gap (window too short) or a liveness stall (window too long). XCE splits textbook Raft's overloaded "election timeout" — `HEARTBEAT_TIMEOUT_MS` (10s) is the leader-failure detector; `ELECTION_TIMEOUT_MS` (5s) is only the candidate vote-collection deadline. Raft's "minimum election timeout" disruption rule maps to HEARTBEAT_TIMEOUT_MS, NOT ELECTION_TIMEOUT_MS. Reusing it keeps stickiness == failure-detect == existing real-vote gate (`raft_cb_vote_eligible`), one predicate; role-gate handles RDHUP/connection-loss for free. A shorter window buys no failover speedup (nobody campaigns before failure-detect anyway; real crashes RDHUP immediately) and only opens a disruption gap on flaky links. Rejected a separate `prevote_stickiness_ms` constant (lets the two timeouts drift). Full comparison table in .kiro/specs/ha-prevote/research.md §6.
|
||||
|
|
@ -130,3 +146,84 @@
|
|||
|
||||
### Decision: full-state rejoin on controller failover (2026-06-16)
|
||||
- On failover, stateful daemons re-join the new primary and apiserver/scheduler receive FULL state (full job list), not a delta. Accepted because failover is rare, some handshake is unavoidable (new primary has no live connection to the daemon until rejoin), and full-state-on-rejoin is simpler/correct and still better than IBM LSF (full-history replace, no snapshot). Lightweight re-attach / delta transfer deferred (YAGNI) — roadmap backlog. Revisit only if measured as a scale problem.
|
||||
|
||||
## Session 2026-06-17 (bug-661 reproduction) — learnings
|
||||
### Do-Not-Repeat
|
||||
- `pkill -9 -f xce_controller` inside an INLINE Bash one-liner kills its OWN shell — the inline command's cmdline contains the literal "xce_controller". Use `pkill -9 -x xce_controller` (exact comm match). Scripts run as a file are safe (cmdline = `bash script.sh ...`).
|
||||
- Daemons spawned with `start.sh ... &` inside a Bash TOOL call get reaped when the call returns. Run multi-process HA E2E via the self-managing script with `run_in_background: true`; write logs to the WORKTREE (e.g. ./dur-run.log), NOT /tmp (the sandbox tears /tmp files down between calls).
|
||||
- The Bash tool has a ~120s wall limit; long foreground HA runs get SIGKILLed and buffered stdout is lost. Use discrete `>>` file appends (each echo flushes) or background+poll.
|
||||
### Key Learnings
|
||||
- bug-661 (committed-job loss on failover) is a RACE: `wal_sync_ms` is hardcoded `DEFAULT_WAL_SYNC_MS=10` (config.h:86), NO CLI/env override. Loss needs the ~10ms un-fsync'd window hit AND a killed+restarted node re-leading before replication catch-up repairs its WAL. Single-node kill+restart soak passed 8 rounds clean; reproduced loss once (4 jobs). Deterministic repro needs a configurable fsync interval (test seam for ha-commit-durability).
|
||||
- Cold-start 3-node election on /tmp/xce-ha-deploy takes ~15s to first CTRL_PRIMARY.
|
||||
- xce_test reads piped stdin via readline and exits on EOF — drive it with `printf 'set masters ..\nclient\njobsub ..\n' | xce_test`.
|
||||
- (2026-06-18, task 1.3 FINAL) bug-661 deterministic repro = `bash src/tools/test-commit-durability.sh --deterministic` (strace-based GAP proof, NOT joblist data-loss). WHY NOT data-loss: single-host SIGKILL is physically NON-deterministic — `persist_wal_async` pwrites INLINE (page cache survives process death; kernel still flushes), and standbys fdatasync-before-WalAck (persist.c:515) so a committed job is always durable off the primary. A rolling-primary-kill data-loss run flips GREEN/RED/RED across identical invocations (REJECTED by review). The DETERMINISTIC slice: with `--wal-sync-ms 30000` the primary's fsync_worker never fires in-window, so the primary replies "jobsubmit ok" with ZERO primary fdatasync between submit and ack. The mode straces each controller (`strace -f -tt -e trace=fdatasync,fsync`, exec-wrap — PTRACE_ATTACH `-p` is blocked by yama, exec-wrap works) and asserts in-window primary syncs==0 ⇒ RED. CONFIRMED 15/15 RED across 5 fresh-cluster runs; asymmetry primary=0 vs standby=2 in-window. Valid RED→GREEN gate: the fsync-before-ack fix (task 2.x) makes the count ≥1 ⇒ GREEN. The exact defect: commit.c:79 `majority_candidate_locked` caps the primary self-contribution at `highest_appended_lsn` (page cache), not a durable_lsn. `--wal-sync-ms` plumbed via deploy-ha.sh start.sh `$*` passthrough.
|
||||
- (2026-06-18, GOTCHA) A SINGLE double-kill (primary + 1 standby) does NOT reproduce bug-661: this codebase enforces STRICT majority — a lone survivor will NOT self-elect (harness log: "no single-voter primary within 40s"). The documented `required=1` loss came from ROLLING failover across terms (incident: ctrl-3 became PRIMARY term=4). Also, committed jobs are durable on the STANDBYS (persist_wal_write_replicated fdatasyncs before WalAck, persist.c:515) — only the PRIMARY's copy is un-fsync'd. So the loss needs rolling hollow re-leads faster than catch-up repair, not just one kill. The deterministic harness drives rolling primary kills to hit this.
|
||||
- (2026-06-18, BUILD) `meson`/`ninja` are NOT on PATH in a fresh sandbox shell — they live in the nix store. Build a single target into the EXISTING `src/xce.build` (no clean) with `cd src && nix develop --command bash -c 'meson compile -C xce.build <target>'`. `xce-build.sh`'s setup_build does `rm -rf` (clean) — avoid it for incremental. The controller/xce_test binaries also need the nix runtime libs (openssl etc.), so run the durability harness itself under `nix develop ./src --command bash -c '...'`.
|
||||
|
||||
## Decision Log — 2026-06-17: ha-commit-durability mechanism
|
||||
- INVARIANT (user, authoritative): reply "ok" to client ⇒ WAL entry fdatasync'd on a fault-tolerant majority (self + ⌊N/2⌋ standbys).
|
||||
- DEFECT: commit.c:majority_candidate_locked counts self as highest_appended_lsn (page-cache), not fsync'd. Standby already persists-before-ACK (persist.c:515); primary does not (relies on fsync_worker timer) → "committed" durable on only 1 node = zero durability fault-tolerance = bug-661.
|
||||
- DECISION: synchronous persistence — eliminate fsync_worker as the durability mechanism. Primary fdatasyncs before its self-contribution covers an LSN; commit.c self uses durable_lsn not highest_appended_lsn; reply stays gated on commit barrier. Liveness-safe because the submit path already blocks OFF the iomgr thread waiting for majority replication (job.c:568-569).
|
||||
- THROUGHPUT: per-op fdatasync; array amplification removed by roadmap item job-array-single-wal-record (array → 1 WAL record → 1 fsync). Group-commit (batched fsync) is a deferred strict optimization (same contract), add only if high-rate independent submits measured as a bottleneck.
|
||||
|
||||
## Decision Log — 2026-06-18: ha-commit-durability design-review reversal (group-commit)
|
||||
- /kiro-validate-design surfaced: highest-volume WAL traffic is WAL_MODE_RETAIN facts (job start/finish/fail/rusage; persist.c wal_classify) — high-frequency, NOT array-collapsible, NOT covered by job-array-single-wal-record, and not client-latency-bound. A pure per-entry synchronous design (delete fsync_worker) would penalize the dominant write path.
|
||||
- REVERSAL (with user): do group-commit NOW. Repurpose fsync_worker timer→demand-driven flusher: on append-signal, one fdatasync over pending appends, advance durable_lsn, wake commit-eval. commit self-count uses durable_lsn. Roadmap "WAL group-commit batching" item ABSORBED into ha-commit-durability.
|
||||
- Lesson: when deciding sync-vs-batched durability, classify ALL write modes first (wal_classify), not just the client-acked submit path. The RETAIN fact stream dominates volume.
|
||||
- Gating items pinned for tasks: (1) freshness-floor formula + raft.tla liveness+safety re-check; (2) FATAL-vs-fence boot decision point; (3) flusher lost-wakeup proof (level-triggered on appended-vs-synced gap).
|
||||
|
||||
## Key Learning — 2026-06-18: commit-ordering rule documented
|
||||
- Authoritative commit/durability invariants now live in src/controller/CLAUDE.md (Safety Invariants + Protocol Verification Checklist):
|
||||
- Commit Ordering (RULE, true today): committed_lsn advances only as a contiguous LSN-ordered prefix; single-writer p->lock is the serialization point; durability batching flushes a contiguous prefix only (durable_lsn is a prefix watermark, never per-entry); log order must preserve causal deps (submit before start); out-of-order/parallel commit NOT supported (single-log architecture).
|
||||
- Commit Durability (TARGET RULE) carries a KNOWN GAP note for bug-661/ha-commit-durability: "committed" does NOT yet imply durable-on-majority until the spec lands. Task 5.3 flips the note to a guarantee when the code lands.
|
||||
- When a safety invariant changes, document it in src/controller/CLAUDE.md (not just the spec) — but mark not-yet-true guarantees as KNOWN GAP so fresh sessions aren't misled into assuming durability that doesn't exist.
|
||||
|
||||
## Key Learning — 2026-06-18: build invocation in worktrees + impl preflight
|
||||
- The flake is at src/flake.nix, NOT the repo root. The xce-build skill's `nix develop --command ...` only works after `cd src`. Correct build: `cd src && nix develop --command bash tools/xce-build.sh controller` (~2-3min, debug+ASAN+tests; nix env is cached, no network). Running from the worktree root fails: "not part of a flake".
|
||||
- meson/ninja are ONLY inside the nix shell (not on PATH); cc/gcc are on PATH but won't configure the meson tree.
|
||||
- ha-commit-durability impl preflight (2026-06-18): tasks approved; cleaned baseline by committing 2 pre-existing WIP units (ha-election stagger + libxce netio error propagation); baseline build+controller tests GREEN. Sandbox caveats stand: 120s Bash cap (background+poll builds), daemon spawns reaped (E2E task 5.2 needs host), no confirmed TLA+ toolchain (tasks 4.1/5.3 need host).
|
||||
|
||||
## Key Learning — 2026-06-18: raft.tla does NOT model durability
|
||||
- raft/raft.tla (Ongaro's spec) restores `log` intact from "stable storage" on restart (raft.tla:165-174) — it ASSUMES durable logs, i.e. assumes exactly what bug-661 violates. Running TLC on the unmodified spec PASSES while the durability bug is still present. raft.tla validates election/freshness logic (mlastLogIndex/LastTerm), NOT commit durability.
|
||||
- The repo has only raft.tla — no .cfg/model/tla2tools.jar/.toolbox/runner; java not on PATH. Project practice = manual conceptual alignment + the CLAUDE.md Protocol Verification Checklist, not automated TLC. So tasks 4.1/5.3 are reasoning tasks (sandbox-doable), not "run TLC" tasks.
|
||||
- TLA+ Toolbox (bundles TLC) — or tla2tools.jar CLI + Java — is sufficient TOOLING for any model-checking, but proving the DURABILITY invariant would require EXTENDING raft.tla (durableLog vs log; Restart truncates to durableLog). That spec-extension is the real work, not the tool. Tasks 4.1/5.3 now carry this caveat so an implementer won't conclude "TLC passed ⇒ durability proven".
|
||||
|
||||
## Key Learning — 2026-06-18: /tmp overlay fdatasync EIO breaks durability tests
|
||||
- This sandbox's `/tmp` can be an OVERLAY filesystem where `fdatasync` returns EIO (errno=5), intermittently under load (3000+/run observed). Controller unit tests `mkdtemp` their WAL dirs into `/tmp/xce_*`, so their fdatasync hits this.
|
||||
- CONSEQUENCE after ha-commit-durability 2.1/2.2/2.4: `committed_lsn` is gated on `durable_lsn` which advances ONLY on fdatasync rc==0 (fail-loud). On EIO, `durable_lsn` correctly HOLDS, so `committed_lsn` cannot advance — this is the gate WORKING AS DESIGNED, but it makes `test_fact_hook_append_is_retained_and_nonblocking` (test_commit_barrier_rollout.c) FAIL on overlay /tmp ("covering ACK must advance committed_lsn, Expected 1 Was 0"). The SAME test FAILS on the clean pre-spec baseline too on overlay /tmp — NOT a regression from any single task.
|
||||
- Prior task reviews reported "0 Failures" because they ran on a tmpfs /tmp or a quiet-IO window (EIO is intermittent). Don't trust a single "0 Failures" run on this fs.
|
||||
- ACTION for final validation (tasks 5.x / /kiro-validate-impl): run the controller suite on a REAL fsync-able filesystem (tmpfs `/dev/shm`, or point test temp dirs at the worktree) so durability tests aren't masked by spurious EIO. The worktree `xce.build` is on a real fs (build is fine); it's the TEST WAL dirs in /tmp that are the problem. Always `rm -rf /tmp/xce_*` before a run (stale dirs also cause an "Expected Non-NULL" mkdtemp cascade).
|
||||
|
||||
## Key Learning — 2026-06-18: meson "Fail:0" masks Unity sub-group failures
|
||||
- The controller suite is ONE meson test ("controller test") running the aggregated `controller_unit_test` binary. A Unity `TEST_ASSERT_*` failure inside a sub-group (e.g. test_fact_hook_append) is recorded + printed `:FAIL:` in testlog.txt but is NOT always propagated to the process exit code, so `meson test` can report `Ok:1 Fail:0` while a test actually FAILED. A C `assert()` (SIGABRT) DOES propagate (so RED-phase asserts are reliable).
|
||||
- RELIABLE GREEN GATE: `grep -c ':FAIL:' src/xce.build/meson-logs/testlog.txt` must be 0 — do NOT trust the meson `Fail:0` summary alone.
|
||||
- Combined with the /tmp-overlay-fdatasync-EIO issue, the documented pre-existing `test_fact_hook_append_is_retained_and_nonblocking` (commit_barrier_rollout) intermittently `:FAIL:`s on overlay /tmp (EIO holds durable_lsn ⇒ committed can't advance over the retained fact — the durability gate working). Final validation (5.3 / /kiro-validate-impl) MUST run on a real fsync-able fs (worktree/tmpfs) AND grep testlog for `:FAIL:`.
|
||||
|
||||
## Decision Log — 2026-06-18: a controller MUST NEVER self-exit (user invariant)
|
||||
- AUTHORITATIVE (user, 2026-06-18): a controller process exits ONLY on (a) a genuine crash (segfault/OOM) or (b) an admin/supervisor kill. It must NEVER self-`exit()`/`abort()` as a *designed response to an operational condition*. Plain `assert()` for genuine programming-invariant bugs is still OK (that's a "crash").
|
||||
- CONSEQUENCE: the ha-commit-durability "FATAL → tlog_exit(); abort()" terminal (tasks 3.3/4.3/4.4, `persist_recover_fatal_if_unrecoverable` + `persist_fatal_hook`) was WRONG — it self-terminated on an operational condition (recovery-truncated WAL after failover). This was the root cause of the 5.2 soak quorum collapse (self-aborting nodes), NOT a sandbox artifact (correcting the earlier cerebrum framing).
|
||||
- CORRECT behavior (user-chosen): an unrecoverable member (truncated, can't catch up) logs LOUDLY (operator-visible ERROR, with the truncation boundary) + stays PERMANENTLY non-promotable/fenced + STAYS ALIVE + KEEPS RETRYING catch-up indefinitely. "Don't serve divergent state" (Req 2.4) = refuse to lead, NEVER kill the process. No bounded-window-to-abort deadline. If a peer returns it catches up and clears the fence; otherwise it waits (loud) for an operator.
|
||||
- REWORK (in progress): remove abort/exit/`persist_fatal_hook` from 3.3; the decision's FATAL terminal becomes a non-terminating "loud permanent fence"; 4.3/4.4 route to it (stay alive); update Req 2.4 / design §3.2 table / §3.3 wording from "FATAL/terminate" to "loud non-promotable fence, stays alive"; update tests.
|
||||
- REWORK DONE (2026-06-18, rework-no-self-exit). Concrete shape: (1) enum `RECOVER_DECISION_FATAL`→`RECOVER_DECISION_FENCE_UNRECOVERABLE` (persist.h:61). (2) `persist_recover_fatal_if_unrecoverable(const inputs*)`→`persist_recover_loud_fence_if_unrecoverable(persist_t* p, const inputs*)` (persist.c:1788) — logs LOUD ERROR (boundary), sets sticky flag, RETURNS; NO tlog_exit/abort. `persist_fatal_hook` global DELETED. (3) sticky `_Atomic bool recovery_unrecoverable` on persist_t (persist.h, after recovery_verdict) + `persist_recovery_unrecoverable(p)` accessor + `persist_clear_recovery_unrecoverable(p)` (clears on a genuine transition + logs "fence RELEASED"); init false in persist_init (persist.c:451). (4) ha.c `ha_evaluate_promotion_fence` FENCE_UNRECOVERABLE arm: calls loud-fence with `p = ha->raft_c ? &ha->raft_c->persist : NULL` (pure-decision callers pass NULL) + sets in_catch_up. (5) ha.c `promotion_fence_timeout_fn` no longer aborts: on still-unrecoverable it RE-EMITS loud alert (sets sticky flag) + RE-ARMS the timer (periodic operator re-alert, indefinite); on caught-up clears in_catch_up + sticky flag; UAF gotcha — capture `t->target_hwm` to a local BEFORE `free(t)` then re-arm (the -Wuse-after-free caught it). `ha_arm_promotion_fence_timer` fail path emits loud-fence once (stays alive), no abort. (6) become_leader gate: also refuses while `persist_recovery_unrecoverable(&c->persist)` (belt-and-suspenders), clears flag on ELIGIBLE. (7) Tests: dropped `persist_fatal_hook`/`dur_fatal_hook`/`g_fatal_hook_calls`; rows 4/4b assert decision==FENCE_UNRECOVERABLE + loud-fence helper RETURNS (process stayed alive) + sticky flag SET + clears on `persist_clear_recovery_unrecoverable`; timer tests use `dur_wait_unrecoverable(c,ms)` polling the sticky flag; renamed *_no_fatal/_prevents_fatal/_bounded_fatal/_defers_fatal. VERIFIED: controller suite Fail:0 / `:FAIL:`==0 ASAN ×2; `grep abort(/exit(/tlog_exit(` ha.c=0 everywhere, persist.c only in comments; production xce_controller `nm` has the 4 new T symbols and NO persist_recover_fatal/persist_fatal_hook. NOTE: `ha_self_restart` (Req 1.3/1.4 demotion execv-restart, ha.h:305) is a SEPARATE primitive — its abort-on-exec-fail is OUT OF SCOPE for this rework (it's the convergence/restart path, not the recovery-unrecoverable operational fence) and was left untouched.
|
||||
|
||||
### xce_test scripting primitives (xce-test-e2e-primitives)
|
||||
- `$jobid`/`$jobcount`/`$?` auto-vars substitute ONLY as a WHOLE token (preceded by line-start/whitespace). `MARK=$jobid` is NOT expanded; use a space: `echo MARK $jobid`. (test.c substitute_autovars, ~line 462)
|
||||
- jobsub on a clean ack sets `$jobid` to the `fmt_jobid` form (non-array `N-N-N-N`, array `N-N-N-N[i]`). To capture it in a shell driver, `echo MARK $jobid` then parse the marker line — do NOT re-impose a numeric id regex (drops array ids).
|
||||
- In a shell driver, `script+=$(printf '...\n')` STRIPS the trailing newline (command-subst), concatenating lines into one bad token. Accumulate with explicit $'\n' instead.
|
||||
- xce_test exit code is the verdict; run with ASAN_OPTIONS=detect_leaks=0 (known client leak) or the exit code is untrustworthy. stderr (errors/sanitizer) does not corrupt stdout capture.
|
||||
|
||||
## Key Learning (2026-06-21) — WAL entry term is dead (bug-851 keystone)
|
||||
`wal_entry_hdr_t.term` (wal_types.h:30) is real but the writer does `hdr.term = p->term` (persist.c:588/726/1040) and **`p->term` is NEVER assigned from `ha->term`** anywhere in the controller — so every WAL entry persists `term=0`. waldump confirms term=0 cluster-wide. This makes Raft (term,lsn) log-matching impossible, so reconcile can't truncate divergent same-LSN tails. Under CHAOTIC failover (kill-the-primary repeatedly), a restarted node recovers a stale term and re-elects at the SAME term (PreVote does not persist/propagate the won term to a majority before the leader acts) → two leaders per term → silent committed-prefix divergence. A CLEAN single failover converges fine (848/849/850 validated). Fixing this is HA-core + persistence = full spec workflow, not a quick patch.
|
||||
|
||||
## Decision Log (2026-06-21) — ha-term-safety: unify config + job state in one log, config bytes out-of-band
|
||||
For the ha-term-safety spec (bug-851 fix), the user chose a SIMPLE, UNIFORM consistency model: configuration changes and job state share ONE replicated (term,lsn) log. A config change is a LIGHTWEIGHT in-log MARKER carrying only a reference (version_tag + sha256), never the bytes. The real config bytes stay OUT of the WAL records and OUT of the cluster (state) snapshot — transferred/stored out-of-band. Design direction: model the config artifact LIKE A SNAPSHOT for replication (chunked, (term,lsn)-anchored, hash-verified, install-before-activate; reuse snapshot_transfer_s pattern), but it is an admin-supplied INPUT, not compacted log state. The marker (not pure snapshot) is mandatory because it gives config the shared (term,lsn) reconcile path; a snapshot anchored at an lsn with no log entry would recreate a separate config reconcile domain (= today's latent config divergence). committed_config_tag becomes DERIVED from the latest committed marker (single commit index). Controller voting-set (peer add/remove) changes are out of scope here (quorum transition → ha-config-bootstrap). Captured as Requirement 7 in .kiro/specs/ha-term-safety/requirements.md.
|
||||
|
||||
## Key Learning (2026-06-21) — raft.c is ELECTION-ONLY; log layer is separate & not TLA+-conformant
|
||||
src/lib/raft/raft.c implements ONLY the election half of raft/raft.tla (Ongaro canonical): currentTerm, votedFor, RequestVote, BecomeLeader, PreVote, UpdateTerm-on-vote-RPCs. It has NO log, NO AppendEntries, NO AdvanceCommitIndex, NO commitIndex. The replicated log = the WAL (persist.c); "AppendEntries" = catch-up streaming/repl.c; "AdvanceCommitIndex" = the commit barrier; conflict reconcile = ha_convergence.c — and NONE of them follow the TLA+ log protocol. Gaps (see .kiro/specs/ha-term-safety/TLA-gap-analysis.md): G1 entry term always 0; G2 replication matches LSN contiguity not prevLogTerm; G3 reconcile truncates by committed-boundary not (term,lsn) conflict; G4 commit is term-blind (no Figure-8 current-term check; own-term noop is partial); G5 standby never adopts leader term from heartbeat/replication (only vote RPCs) → term reuse across restart → two leaders/term; G6 vote freshness is LSN-only not (lastLogTerm,lastLogIndex). G5 confirmed live: rejoined ctrl-1 reported term:2 under a term-3 primary. These compose into bug-851. Config is a THIRD domain entirely outside both (committed_config_tag, no term).
|
||||
|
||||
## Decision Log (2026-06-21) — make src/lib/raft strictly implement raft.tla (library owns the log)
|
||||
User chose to EXTEND the raft library to strictly follow raft/raft.tla, using the verified spec as a conformance oracle, instead of patching the controller's hand-rolled log protocol. Scaffolded via roadmap + /kiro-spec-batch as the "Raft TLA+ Conformance" series (3 specs, all generated+auto-approved 2026-06-21, cross-spec review PASSED):
|
||||
- raft-log-replication (8 reqs/24 tasks): library owns log; per-entry term (G1), AppendEntries prevLogTerm match (G2), (term,index) conflict truncation (G3), UpdateTerm on EVERY RPC (G5). New callbacks: on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from. Entry pts: raft_log_append, raft_on_append_entries_request/reply, raft_send_append_entries. State: log_len, next_index[], match_index[]. Index 1-based = WAL LSN.
|
||||
- raft-commit-safety (6 reqs/18 tasks): AdvanceCommitIndex Figure-8 current-term rule (G4); (lastLogTerm,lastLogIndex) freshness (G6). New: raft_advance_commit, on_commit_advanced(commit_index) [TRIGGERS the existing barrier, never replaces], commit_index state. raft_on_vote_request/raft_on_pre_vote_request gain (cand_last_log_term, cand_last_log_index); library now owns logOk (locus moved from caller/dispatcher → library; supersedes raft.h:246-248 doc).
|
||||
- raft-sim-conformance (8 reqs/23 tasks): extend raft_sim.c into in-process oracle driving raft.tla Next-actions, asserting the 4 safety invariants; negative test via compile-time RAFT_SIM_BREAK_* switches (test build only). Would have caught bug-851.
|
||||
Pure library core (no I/O/threads) + controller callback adapters keeps it simulator-verifiable. Preserve as policy: PreVote, priority leadership, sync commit-barrier + persist-before-ACK durability, controller-never-self-terminates. INTENTIONAL safe deviation from raft.tla: conflict branch does truncate-THEN-append in one handler (raft.tla is truncate-only) — documented. ha-term-safety reframed = controller wiring + config R7/R8 + live regression; its R1-R6 realized by the library specs. Implement order: raft-library(done) → log-replication → commit-safety → sim-conformance, then ha-term-safety.
|
||||
|
|
|
|||
2296
.wolf/memory.md
2296
.wolf/memory.md
File diff suppressed because it is too large
Load diff
107
package-lock.json
generated
Normal file
107
package-lock.json
generated
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
{
|
||||
"name": "persistent_ha",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"dependencies": {
|
||||
"@colbymchenry/codegraph": "^0.9.9"
|
||||
}
|
||||
},
|
||||
"node_modules/@colbymchenry/codegraph": {
|
||||
"version": "0.9.9",
|
||||
"resolved": "https://registry.npmjs.org/@colbymchenry/codegraph/-/codegraph-0.9.9.tgz",
|
||||
"integrity": "sha512-23Dl9q0RHKVhJRp+Y823GfmQJjcPiRycoKN7TkaI8eJitluRvieoVlclPILqua8znvkAw9KevYziFfK3k74CBQ==",
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
"codegraph": "npm-shim.js"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@colbymchenry/codegraph-darwin-arm64": "0.9.9",
|
||||
"@colbymchenry/codegraph-darwin-x64": "0.9.9",
|
||||
"@colbymchenry/codegraph-linux-arm64": "0.9.9",
|
||||
"@colbymchenry/codegraph-linux-x64": "0.9.9",
|
||||
"@colbymchenry/codegraph-win32-arm64": "0.9.9",
|
||||
"@colbymchenry/codegraph-win32-x64": "0.9.9"
|
||||
}
|
||||
},
|
||||
"node_modules/@colbymchenry/codegraph-darwin-arm64": {
|
||||
"version": "0.9.9",
|
||||
"resolved": "https://registry.npmjs.org/@colbymchenry/codegraph-darwin-arm64/-/codegraph-darwin-arm64-0.9.9.tgz",
|
||||
"integrity": "sha512-Mu6xhG4bF1OLeTqN2eSI5U8z2esymK/RjxP3T8cCigA4YprDIvd3XSGmTZREbDNRCuqrCi0R1LCJ7fsdFfpAPA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
]
|
||||
},
|
||||
"node_modules/@colbymchenry/codegraph-darwin-x64": {
|
||||
"version": "0.9.9",
|
||||
"resolved": "https://registry.npmjs.org/@colbymchenry/codegraph-darwin-x64/-/codegraph-darwin-x64-0.9.9.tgz",
|
||||
"integrity": "sha512-xShaChiPJsajmBxvr1yP2XWsYstq0kJzHwQmiUG1huWrE14tP9NJnJ36hsbzHfHiqwBOJo2+5MZHuXmmzJa/vQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
]
|
||||
},
|
||||
"node_modules/@colbymchenry/codegraph-linux-arm64": {
|
||||
"version": "0.9.9",
|
||||
"resolved": "https://registry.npmjs.org/@colbymchenry/codegraph-linux-arm64/-/codegraph-linux-arm64-0.9.9.tgz",
|
||||
"integrity": "sha512-zccx1m3gGB2jAGfz41U+1GQNK12mtLGuX48XyZXiyQIGFsJQCFS8v/OvShAfPygSGhUr3zKMetbkMnztp7ULtg==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
]
|
||||
},
|
||||
"node_modules/@colbymchenry/codegraph-linux-x64": {
|
||||
"version": "0.9.9",
|
||||
"resolved": "https://registry.npmjs.org/@colbymchenry/codegraph-linux-x64/-/codegraph-linux-x64-0.9.9.tgz",
|
||||
"integrity": "sha512-vQfe5VdSQb/cVo1pwO12FyzTNaDIXtN3dLuAS/LIPsg2tQVyT9fwr10QmL10qWUNBShjgnRsomHdMG7hCjW3Yg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
]
|
||||
},
|
||||
"node_modules/@colbymchenry/codegraph-win32-arm64": {
|
||||
"version": "0.9.9",
|
||||
"resolved": "https://registry.npmjs.org/@colbymchenry/codegraph-win32-arm64/-/codegraph-win32-arm64-0.9.9.tgz",
|
||||
"integrity": "sha512-U2QlTT434ulbEw6sTJVS+VNhjJUbZqCnw8tXuXSKkcuzGPZq/Hxoofe9UpY77+J9hAJu9PwwiWjEEaiDX9yb1w==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
]
|
||||
},
|
||||
"node_modules/@colbymchenry/codegraph-win32-x64": {
|
||||
"version": "0.9.9",
|
||||
"resolved": "https://registry.npmjs.org/@colbymchenry/codegraph-win32-x64/-/codegraph-win32-x64-0.9.9.tgz",
|
||||
"integrity": "sha512-N7Xgt80BeEy78nNeqHaUgcsngxmy1Hty570QSa4hOh4zl6z6v6n7i9ZSUfraktESVUkfmMbUwoqfEzVwUbB0/g==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
1
package.json
Normal file
1
package.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{}
|
||||
|
|
@ -1356,6 +1356,20 @@ int cluster_init_phase2(cluster_t* c)
|
|||
return rc;
|
||||
}
|
||||
|
||||
/* bug-848: a (re)started member recovered its own WAL above — make it
|
||||
* ADVERTISE that recovered position for catch-up (sr->applied_lsn), not the
|
||||
* ha_init default of 0. Otherwise the primary streams the whole log and the
|
||||
* receiver overwrites already-consistent records instead of fetching only
|
||||
* the missing suffix (Raft incremental catch-up). */
|
||||
ha_sync_applied_lsn_from_recovery(&c->ha, &c->persist);
|
||||
INFO("HA: persist_recover done — recovered durable_floor=%" PRIu64
|
||||
" current_lsn=%" PRIu64 "; applied_lsn=%" PRIu64
|
||||
" advertised for catch-up",
|
||||
persist_durable_floor(&c->persist),
|
||||
(uint64_t)c->persist.current_lsn,
|
||||
(uint64_t)atomic_load_explicit(&c->ha.applied_lsn,
|
||||
memory_order_acquire));
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1570,6 +1570,16 @@ static int ha_apply_single_entry(persist_t *p, wal_catchup_entry_t *e)
|
|||
int prc = persist_wal_write_replicated(p, e->lsn, (uint32_t)e->entry_type,
|
||||
e->raw_payload,
|
||||
(uint32_t)e->raw_payload_len);
|
||||
if (prc == -EAGAIN) {
|
||||
/* Contiguity guard (bug-850): this entry is ahead of our log tail
|
||||
* (a forward gap) — e.g. a live broadcast that arrived before the
|
||||
* in-order catch-up of the prefix. NOT an error: drop it quietly and do
|
||||
* NOT ack, so the catch-up continuation re-streams from current_lsn and
|
||||
* the log fills in order. */
|
||||
TRACE("ha_apply_single_entry: lsn=%" PRIu64 " ahead of log tail — "
|
||||
"dropped (will arrive via in-order catch-up)", e->lsn);
|
||||
return prc;
|
||||
}
|
||||
if (prc != 0) {
|
||||
ERROR("ha_apply_single_entry: persist_wal_write_replicated failed for "
|
||||
"lsn=%" PRIu64 " type=%u rc=%d — not applying/acking",
|
||||
|
|
@ -1636,11 +1646,14 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
|
|||
|
||||
int rc = ha_apply_single_entry(p, sub);
|
||||
if (rc != 0) {
|
||||
ERROR("ha_apply_wal_catchup_entry: apply_wal_entry failed "
|
||||
"for batch index %d lsn=%" PRIu64 " type=%u rc=%d "
|
||||
"— skipping entry",
|
||||
i, sub->lsn, (unsigned)sub->entry_type, rc);
|
||||
/* Task 3.2: no ACK for failed entries */
|
||||
/* -EAGAIN = contiguity gap (bug-850): entry ahead of our tail,
|
||||
* dropped quietly — not an error (catch-up will backfill). */
|
||||
if (rc != -EAGAIN)
|
||||
ERROR("ha_apply_wal_catchup_entry: apply_wal_entry failed "
|
||||
"for batch index %d lsn=%" PRIu64 " type=%u rc=%d "
|
||||
"— skipping entry",
|
||||
i, sub->lsn, (unsigned)sub->entry_type, rc);
|
||||
/* Task 3.2: no ACK for failed/dropped entries */
|
||||
continue;
|
||||
}
|
||||
|
||||
|
|
@ -1662,9 +1675,12 @@ int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
|
|||
/* ── Single-entry mode (task 2.1 backward-compatible path) ────────────── */
|
||||
int rc = ha_apply_single_entry(p, entry);
|
||||
if (rc != 0) {
|
||||
ERROR("ha_apply_wal_catchup_entry: apply_wal_entry failed for "
|
||||
"lsn=%" PRIu64 " type=%u rc=%d — skipping entry",
|
||||
entry->lsn, (unsigned)entry->entry_type, rc);
|
||||
/* -EAGAIN = contiguity gap (bug-850): dropped quietly, backfilled by
|
||||
* the in-order catch-up; any other rc is a real failure. */
|
||||
if (rc != -EAGAIN)
|
||||
ERROR("ha_apply_wal_catchup_entry: apply_wal_entry failed for "
|
||||
"lsn=%" PRIu64 " type=%u rc=%d — skipping entry",
|
||||
entry->lsn, (unsigned)entry->entry_type, rc);
|
||||
/* Task 3.2: no ACK on apply failure */
|
||||
return rc;
|
||||
}
|
||||
|
|
@ -2600,6 +2616,23 @@ uint64_t ha_candidate_advertise_lsn(persist_t *p)
|
|||
return persist_durable_floor(p);
|
||||
}
|
||||
|
||||
/* bug-848: see ha.h. After local WAL recovery, advertise the recovered durable
|
||||
* tip for catch-up so the primary streams only the missing suffix instead of
|
||||
* re-streaming + overwriting consistent records. Monotonic: never lowers a
|
||||
* higher applied_lsn (e.g. if catch-up already advanced it). */
|
||||
void ha_sync_applied_lsn_from_recovery(ha_state_t *ha, persist_t *p)
|
||||
{
|
||||
if (!ha || !p)
|
||||
return;
|
||||
|
||||
uint64_t recovered = persist_durable_floor(p);
|
||||
uint64_t cur = atomic_load_explicit(&ha->applied_lsn,
|
||||
memory_order_acquire);
|
||||
if (recovered > cur)
|
||||
atomic_store_explicit(&ha->applied_lsn, recovered,
|
||||
memory_order_release);
|
||||
}
|
||||
|
||||
/* ── Vote Handler (Req 2.2–2.6, 6.1, 6.2, 6.4, 9.1) ────────────────────────
|
||||
*
|
||||
* handle_request_vote: process an incoming OP_requestVote RPC.
|
||||
|
|
|
|||
|
|
@ -793,6 +793,21 @@ void ha_vote_self_freshness(ha_state_t *ha,
|
|||
*/
|
||||
uint64_t ha_candidate_advertise_lsn(persist_t *p);
|
||||
|
||||
/* ha_sync_applied_lsn_from_recovery (bug-848) — after local WAL recovery on
|
||||
* (re)start, set ha->applied_lsn to the position the member actually recovered
|
||||
* (persist_durable_floor(p)), instead of the ha_init default of 0.
|
||||
*
|
||||
* Why: a restarted standby advertises ha->applied_lsn to the primary in its
|
||||
* state report; the primary then streams the catch-up range [applied_lsn+1 ..
|
||||
* committed]. If a node that already holds [1..N] on disk reports 0, the
|
||||
* primary re-streams the WHOLE log and the receiver O_TRUNC-overwrites already-
|
||||
* consistent records (instead of Raft incremental catch-up of only the missing
|
||||
* suffix). This mirrors the value the election/PreVote path already trusts
|
||||
* (ha_candidate_advertise_lsn == persist_durable_floor), so the catch-up and
|
||||
* vote-freshness advertisements finally agree. NULL-safe; never lowers an
|
||||
* already-higher applied_lsn (monotonic). */
|
||||
void ha_sync_applied_lsn_from_recovery(ha_state_t *ha, persist_t *p);
|
||||
|
||||
/*
|
||||
* ha_elect_evaluate_and_signal: evaluate accumulated ha_elect_ctx_t votes.
|
||||
* Sets ctx->result (1=primary, 0=standby) and posts ctx->done semaphore.
|
||||
|
|
|
|||
|
|
@ -53,7 +53,7 @@ controller = executable('xce_controller',
|
|||
link_args: wrap_link_args,
|
||||
install: true)
|
||||
|
||||
xce_test_src = [ 'testsuite/test.c']
|
||||
xce_test_src = [ 'testsuite/test.c', 'testsuite/waldump.c']
|
||||
|
||||
readline_dep = dependency('readline')
|
||||
ncurses_dep = dependency('ncurses')
|
||||
|
|
@ -67,6 +67,8 @@ xce_test_deps = [
|
|||
history_dep,
|
||||
ncurses_dep,
|
||||
tinylog_dep,
|
||||
zlib_dep, # waldump: crc32 WAL validation
|
||||
c_capnproto_dep, # waldump: decode JOB_SUBMIT jobid
|
||||
]
|
||||
xce_test = executable('xce_test',
|
||||
xce_test_src,
|
||||
|
|
@ -136,6 +138,8 @@ controller_unit_test_src = [
|
|||
'testsuite/test_ctrlstatus.c',
|
||||
'testsuite/test_ha_heartbeat_committed.c',
|
||||
'testsuite/test_ha_vote_floor_heartbeat.c',
|
||||
'testsuite/test_waldump.c',
|
||||
'testsuite/waldump.c',
|
||||
'ctrlstatus.c',
|
||||
'commit.c',
|
||||
'job.c',
|
||||
|
|
|
|||
|
|
@ -317,6 +317,31 @@ skip_election:;
|
|||
if (role == CTRL_STANDBY || role == CTRL_INIT) {
|
||||
atomic_store_explicit(&c.ha.role, (int32_t)CTRL_STANDBY,
|
||||
memory_order_release);
|
||||
|
||||
/* bug-849: a follower that REJOINS an existing cluster skips its own
|
||||
* election (a primary already exists), so the election-result path
|
||||
* that calls cluster_init_phase2 — which builds resources, runs
|
||||
* persist_recover, and (bug-848) advertises the recovered
|
||||
* applied_lsn — NEVER runs on this path. The node then comes up on
|
||||
* an un-recovered WAL (current_lsn=1, applied_lsn=0), the primary
|
||||
* re-streams the whole log over its consistent records, and the
|
||||
* committed prefix is lost. Run phase2 HERE too (idempotent via
|
||||
* phase2_done, same once-per-process contract as the election
|
||||
* callers) so EVERY boot path recovers its own WAL and advertises
|
||||
* its real position BEFORE cluster_run starts accepting replication.
|
||||
* Recovery is decoupled from the primary's config push: a restarted
|
||||
* node already loaded its config from disk pre-election. */
|
||||
if (!c.ha.phase2_done) {
|
||||
int prc = cluster_init_phase2(&c);
|
||||
if (prc != 0) {
|
||||
WARN("myapp_run: cluster_init_phase2 on standby rejoin "
|
||||
"failed (rc=%d) — WAL not recovered; falling back to "
|
||||
"election-path recovery", prc);
|
||||
} else {
|
||||
c.ha.phase2_done = 1;
|
||||
}
|
||||
}
|
||||
|
||||
INFO("myapp_run: CTRL_STANDBY — waiting for primary to connect and push config");
|
||||
/* Arm heartbeat watchdog even when we skipped the election.
|
||||
* If the known primary never connects within heartbeat_timeout_ms,
|
||||
|
|
|
|||
|
|
@ -688,6 +688,23 @@ int persist_wal_write_replicated(persist_t* p, uint64_t lsn, uint32_t type,
|
|||
return 0;
|
||||
}
|
||||
|
||||
/* Contiguity guard (bug-850): REFUSE a forward GAP. current_lsn is the
|
||||
* next-expected LSN, so a replicated entry with lsn > current_lsn would
|
||||
* leave [current_lsn .. lsn-1] missing AND advance current_lsn past that
|
||||
* hole — after which the idempotency guard above silently DROPS the later
|
||||
* in-order catch-up of the prefix (the failover prefix-loss mechanism).
|
||||
* Append only the contiguous next entry (lsn == current_lsn); reject a gap
|
||||
* without writing or acking so the ACK-driven catch-up continuation
|
||||
* re-streams from current_lsn and the log fills in order. This is Raft
|
||||
* log-matching: append entry N only when the log already holds N-1. A
|
||||
* behind standby that keeps receiving live broadcasts simply drops the
|
||||
* out-of-order ones until catch-up backfills the gap (no loss). -EAGAIN is
|
||||
* the "retry via catch-up" signal the apply path treats as a quiet skip. */
|
||||
if (lsn > p->current_lsn) {
|
||||
pthread_mutex_unlock(&p->lock);
|
||||
return -EAGAIN;
|
||||
}
|
||||
|
||||
/* Open first segment lazily. open_new_segment names the file from
|
||||
* current_lsn — for a fresh standby receiving lsn 1 first, current_lsn is
|
||||
* 1 (reserve-lsn-0), so the segment is seg-...001.wal. */
|
||||
|
|
|
|||
|
|
@ -63,6 +63,7 @@ extern void test_commit_durability(void);
|
|||
extern void test_ctrlstatus(void);
|
||||
extern void test_ha_heartbeat_committed(void);
|
||||
extern void test_ha_vote_floor_heartbeat(void);
|
||||
extern void test_waldump(void);
|
||||
|
||||
void setUp(void) { return; }
|
||||
|
||||
|
|
@ -195,6 +196,7 @@ int main(int argc, char* argv[])
|
|||
TEST(test_ctrlstatus);
|
||||
TEST(test_ha_heartbeat_committed);
|
||||
TEST(test_ha_vote_floor_heartbeat);
|
||||
TEST(test_waldump);
|
||||
|
||||
return UNITY_END();
|
||||
}
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@
|
|||
#include "delay.h"
|
||||
#include "idgen.h"
|
||||
#include "preason.h"
|
||||
#include "waldump.h" /* offline WAL inspector (cmd_waldump) */
|
||||
#include "xce/cluster.h" /* xce_parse_peers, xce_cluster_config_t */
|
||||
#include "xce/types.h"
|
||||
|
||||
|
|
@ -122,6 +123,7 @@ extern int cmd_expect_jobs(int, char*[]);
|
|||
extern int cmd_expect_job(int, char*[]);
|
||||
extern int cmd_wait_primary(int, char*[]);
|
||||
extern int cmd_ctrlstatus(int, char*[]);
|
||||
extern int cmd_waldump(int, char*[]);
|
||||
static int worker_house_keeping(void*);
|
||||
|
||||
/* Session verdict / loop state (xce-test-e2e-primitives task 1.1).
|
||||
|
|
@ -179,6 +181,7 @@ static command_t top_commands[] = {
|
|||
{ "client", cmd_client, "In client context" },
|
||||
{ "set", cmd_set, "Show/Set settings" },
|
||||
{ "echo", cmd_echo, "Print a message" },
|
||||
{ "waldump", cmd_waldump, "Inspect WAL segments" },
|
||||
{ "quit", cmd_quit, "Quit xce test" },
|
||||
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL }
|
||||
};
|
||||
|
|
@ -216,6 +219,7 @@ static command_t client_commands[] = {
|
|||
{ "wait-primary", cmd_wait_primary,
|
||||
"wait until a primary serves [timeout_ms]" },
|
||||
{ "ctrlstatus", cmd_ctrlstatus, "show controller/cluster status" },
|
||||
{ "waldump", cmd_waldump, "Inspect WAL segments" },
|
||||
{ "shutdown", cmd_shutdown, "shutdown xce controller" },
|
||||
{ "jobrun", cmd_jobrun, "run a job" },
|
||||
{ "jobsub", cmd_jobsub, "submit a job" },
|
||||
|
|
@ -683,6 +687,35 @@ int cmd_echo(int argc, char* argv[])
|
|||
return 0;
|
||||
}
|
||||
|
||||
/* waldump <path> [<path> ...] — offline WAL inspector for failover/durability
|
||||
* triage. Each path may be a seg-*.wal file, a wal/ dir, or a node/data dir.
|
||||
* Read-only: lists every entry, decodes JOB_SUBMIT jobids, and flags LSN gaps
|
||||
* (a missing committed prefix is THE failover job-loss signal) plus the CRC
|
||||
* recovered tip. Returns nonzero if any path failed or any gap was found, so
|
||||
* it doubles as an assertion in scripted E2E runs. */
|
||||
int cmd_waldump(int argc, char* argv[])
|
||||
{
|
||||
if (argc < 2) {
|
||||
fprintf(stderr,
|
||||
"usage: waldump <wal_dir|data_dir|seg_file> [more...]\n");
|
||||
return -1;
|
||||
}
|
||||
|
||||
int rc = 0;
|
||||
for (int i = 1; i < argc; i++) {
|
||||
waldump_stats_t st;
|
||||
int r = waldump_path(argv[i], stdout, &st);
|
||||
if (r != 0) {
|
||||
fprintf(stderr, "waldump: %s: %s\n", argv[i], strerror(-r));
|
||||
rc = -1;
|
||||
continue;
|
||||
}
|
||||
if (st.n_gaps > 0 || st.crc_broken)
|
||||
rc = -1; /* surface the defect as a nonzero command verdict */
|
||||
}
|
||||
return rc;
|
||||
}
|
||||
|
||||
int cmd_up(int argc, char* argv[])
|
||||
{
|
||||
if (active_ctx->parent != NULL) {
|
||||
|
|
|
|||
|
|
@ -440,8 +440,23 @@ static void test_fact_hook_append_is_retained_and_nonblocking(void)
|
|||
"the uncommitted fact must be pending (lsn > committed_lsn), "
|
||||
"never confirmed early");
|
||||
|
||||
/* ...and a covering ACK later commits the retained entry — engine
|
||||
* accounting advances committed_lsn over it, no redelivery needed */
|
||||
/* ha-commit-durability 2.2: the primary's self-contribution to the commit
|
||||
* majority is now capped at durable_lsn, which the demand-driven
|
||||
* group-commit flusher advances asynchronously after fdatasync. Force the
|
||||
* fact durable here (synchronous fdatasync) so the covering ACK below
|
||||
* deterministically completes the majority (durable self + standby ACK);
|
||||
* a bare synchronous read would otherwise race the flusher and observe
|
||||
* durable_lsn == 0 → self-cap 0 → committed_lsn unchanged. This mirrors the
|
||||
* durable_lsn priming done for the in-memory stub fixtures in
|
||||
* barrier_setup_vcpus. */
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, persist_flush(g_fp),
|
||||
"fact must fdatasync (durable) before the covering ACK");
|
||||
TEST_ASSERT_TRUE_MESSAGE(persist_durable_lsn(g_fp) >= fact_lsn,
|
||||
"durable_lsn must cover the fact after persist_flush");
|
||||
|
||||
/* ...and a covering ACK then commits the now-durable retained entry —
|
||||
* repl_on_ack drives commit_on_ack synchronously, advancing committed_lsn
|
||||
* over it, no redelivery needed */
|
||||
repl_on_ack(&g_fp->repl, &g_fep_a, fact_lsn);
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
|
||||
fact_lsn, atomic_load(&g_fp->committed_lsn),
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@
|
|||
#include <unity/unity.h>
|
||||
|
||||
#include "config.h"
|
||||
#include "ha.h" /* ha_state_t, ha_sync_applied_lsn_from_recovery (bug-848) */
|
||||
#include "persist.h"
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
|
@ -248,6 +249,66 @@ static void test_recover_picks_latest_snapshot(void)
|
|||
/* Test suite entry point */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* bug-848: a recovered member must advertise its REAL last LSN, not 0 */
|
||||
/* (else the primary re-streams the whole log and overwrites consistent */
|
||||
/* records instead of fetching only the missing suffix). */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
static void test_recovered_member_advertises_real_applied_lsn(void)
|
||||
{
|
||||
char tmp[256];
|
||||
strcpy(tmp, "/tmp/xce_rec848_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||
|
||||
/* Phase 1: write several WAL entries, flush, stop (simulate a node that
|
||||
* served writes, then was killed). */
|
||||
{
|
||||
persist_t* p = make_fresh_persist(tmp);
|
||||
TEST_ASSERT_NOT_NULL(p);
|
||||
for (int i = 0; i < 5; i++)
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
persist_wal_async(p, WAL_JOB_UPDATE, "z", 1));
|
||||
TEST_ASSERT_EQUAL_INT(0, persist_flush(p));
|
||||
teardown_persist(p);
|
||||
}
|
||||
|
||||
/* Phase 2: restart — fresh persist recovers the WAL; a freshly ha-init'd
|
||||
* member (applied_lsn == 0) must be synced to the recovered durable tip. */
|
||||
{
|
||||
persist_t* p2 = make_fresh_persist(tmp);
|
||||
TEST_ASSERT_NOT_NULL(p2);
|
||||
TEST_ASSERT_EQUAL_INT(0, persist_recover(p2));
|
||||
|
||||
uint64_t recovered_floor = persist_durable_floor(p2);
|
||||
TEST_ASSERT_TRUE(recovered_floor > 0); /* WAL was non-empty */
|
||||
|
||||
ha_state_t ha;
|
||||
memset(&ha, 0, sizeof(ha));
|
||||
atomic_store_explicit(&ha.applied_lsn, 0ULL, memory_order_release);
|
||||
|
||||
ha_sync_applied_lsn_from_recovery(&ha, p2);
|
||||
|
||||
uint64_t advertised =
|
||||
atomic_load_explicit(&ha.applied_lsn, memory_order_acquire);
|
||||
/* The bug: this used to stay 0 → primary re-streams [1..committed]. */
|
||||
TEST_ASSERT_TRUE(advertised > 0);
|
||||
TEST_ASSERT_EQUAL_UINT64(recovered_floor, advertised);
|
||||
|
||||
/* Monotonic: never lowers an already-higher applied_lsn. */
|
||||
atomic_store_explicit(&ha.applied_lsn, recovered_floor + 100,
|
||||
memory_order_release);
|
||||
ha_sync_applied_lsn_from_recovery(&ha, p2);
|
||||
TEST_ASSERT_EQUAL_UINT64(
|
||||
recovered_floor + 100,
|
||||
atomic_load_explicit(&ha.applied_lsn, memory_order_acquire));
|
||||
|
||||
teardown_persist(p2);
|
||||
}
|
||||
|
||||
printf("test_recovered_member_advertises_real_applied_lsn: PASS\n");
|
||||
}
|
||||
|
||||
void test_persist_recovery(void)
|
||||
{
|
||||
RUN_TEST(test_recover_from_empty_state);
|
||||
|
|
@ -255,4 +316,5 @@ void test_persist_recovery(void)
|
|||
RUN_TEST(test_recover_replays_wal_after_checkpoint);
|
||||
RUN_TEST(test_recover_rejects_corrupt_snapshot);
|
||||
RUN_TEST(test_recover_picks_latest_snapshot);
|
||||
RUN_TEST(test_recovered_member_advertises_real_applied_lsn);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -395,6 +395,50 @@ static void test_repl_write_dedup_idempotent(void)
|
|||
printf("test_repl_write_dedup_idempotent: PASS\n");
|
||||
}
|
||||
|
||||
/* bug-850 contiguity guard: a replicated entry whose lsn is BEYOND the next-
|
||||
* expected current_lsn (a forward gap) must be REJECTED with -EAGAIN without
|
||||
* writing or advancing current_lsn — otherwise it would advance the watermark
|
||||
* past the hole and the later in-order catch-up of the prefix would be silently
|
||||
* dropped by the idempotency guard (the failover prefix-loss mechanism). The
|
||||
* entry becomes writable only once the gap is filled contiguously. */
|
||||
static void test_repl_write_rejects_forward_gap(void)
|
||||
{
|
||||
char tmpdir[256];
|
||||
persist_t* p = make_test_persist_in(tmpdir, sizeof(tmpdir));
|
||||
|
||||
/* Contiguous first entry: lsn 1 == current_lsn → written. */
|
||||
TEST_ASSERT_EQUAL_INT(0, persist_wal_write_replicated(
|
||||
p, 1, WAL_JOB_UPDATE, "a", 1));
|
||||
TEST_ASSERT_EQUAL_UINT64(2, p->current_lsn);
|
||||
|
||||
/* Forward gap: lsn 5 > current_lsn(2) — like a live broadcast arriving
|
||||
* ahead of catch-up. Must be refused (-EAGAIN) and leave current_lsn put. */
|
||||
int rc_gap = persist_wal_write_replicated(p, 5, WAL_JOB_UPDATE, "e", 1);
|
||||
TEST_ASSERT_EQUAL_INT(-EAGAIN, rc_gap);
|
||||
TEST_ASSERT_EQUAL_UINT64(2, p->current_lsn); /* did NOT jump past the gap */
|
||||
|
||||
/* Fill the gap in order: 2,3,4 contiguous → accepted. */
|
||||
for (uint64_t lsn = 2; lsn <= 4; lsn++)
|
||||
TEST_ASSERT_EQUAL_INT(0, persist_wal_write_replicated(
|
||||
p, lsn, WAL_JOB_UPDATE, "x", 1));
|
||||
TEST_ASSERT_EQUAL_UINT64(5, p->current_lsn);
|
||||
|
||||
/* Now lsn 5 is contiguous (== current_lsn) → accepted. */
|
||||
TEST_ASSERT_EQUAL_INT(0, persist_wal_write_replicated(
|
||||
p, 5, WAL_JOB_UPDATE, "e", 1));
|
||||
TEST_ASSERT_EQUAL_UINT64(6, p->current_lsn);
|
||||
|
||||
destroy_test_persist(p);
|
||||
|
||||
/* Recovery yields a clean contiguous chain [1..5] → current_lsn=6. */
|
||||
persist_t* p2 = reopen_test_persist(tmpdir);
|
||||
TEST_ASSERT_TRUE(persist_recover(p2) >= 0);
|
||||
TEST_ASSERT_EQUAL_UINT64(6, p2->current_lsn);
|
||||
destroy_test_persist(p2);
|
||||
|
||||
printf("test_repl_write_rejects_forward_gap: PASS\n");
|
||||
}
|
||||
|
||||
/* WAL_CHECKPOINT must never be persisted via the replicated-write path
|
||||
* (standbys never receive it; it must be rejected, not written). */
|
||||
static void test_repl_write_rejects_checkpoint(void)
|
||||
|
|
@ -428,5 +472,6 @@ void test_repl_apply(void)
|
|||
RUN_TEST(test_repl_write_persists_and_recovers);
|
||||
RUN_TEST(test_repl_write_crc_chain_recovers);
|
||||
RUN_TEST(test_repl_write_dedup_idempotent);
|
||||
RUN_TEST(test_repl_write_rejects_forward_gap);
|
||||
RUN_TEST(test_repl_write_rejects_checkpoint);
|
||||
}
|
||||
|
|
|
|||
295
src/controller/testsuite/test_waldump.c
Normal file
295
src/controller/testsuite/test_waldump.c
Normal file
|
|
@ -0,0 +1,295 @@
|
|||
#define _GNU_SOURCE
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <zlib.h>
|
||||
|
||||
#include <unity/unity.h>
|
||||
|
||||
#include "config.h" /* WAL_HDR_CRC_SIZE, WAL_CRC_NONE */
|
||||
#include "controller.capnp.h" /* Job encode for the JOB_SUBMIT decode test */
|
||||
#include "wal_types.h"
|
||||
#include "waldump.h"
|
||||
|
||||
/* ------------------------------------------------------------------ *
|
||||
* Test harness: hand-write raw WAL segments so the test fully controls
|
||||
* LSNs (to inject gaps), the CRC chain (to inject a break), and a real
|
||||
* Cap'n Proto Job payload (to exercise the submit jobid decode). This
|
||||
* also pins the on-disk WAL format independently of persist.c.
|
||||
* ------------------------------------------------------------------ */
|
||||
|
||||
static char g_dir[256];
|
||||
|
||||
/* Append one entry to `fp`, chaining prev_crc from *last_crc and updating it.
|
||||
* Mirrors persist.c's append: CRC32 over WAL_HDR_CRC_SIZE header bytes + payload. */
|
||||
static void put_entry(FILE* fp, uint64_t term, uint64_t lsn, uint32_t type,
|
||||
const void* payload, uint32_t plen, uint32_t* last_crc,
|
||||
int first_in_seg)
|
||||
{
|
||||
wal_entry_hdr_t hdr;
|
||||
memset(&hdr, 0, sizeof(hdr));
|
||||
hdr.term = term;
|
||||
hdr.lsn = lsn;
|
||||
hdr.timestamp = 0;
|
||||
hdr.type = type;
|
||||
hdr.payload_len = plen;
|
||||
hdr.prev_crc = first_in_seg ? WAL_CRC_NONE : *last_crc;
|
||||
|
||||
uint32_t c = crc32(0L, (const Bytef*)&hdr, WAL_HDR_CRC_SIZE);
|
||||
if (plen > 0)
|
||||
c = crc32(c, (const Bytef*)payload, plen);
|
||||
hdr.crc = c;
|
||||
*last_crc = c;
|
||||
|
||||
fwrite(&hdr, sizeof(hdr), 1, fp);
|
||||
if (plen > 0)
|
||||
fwrite(payload, plen, 1, fp);
|
||||
}
|
||||
|
||||
/* Encode a minimal Cap'n Proto Job carrying `jobid`+`subtime` (what a real
|
||||
* WAL_JOB_SUBMIT payload looks like). Returns malloc'd bytes; caller frees.
|
||||
* subtime lets a test make two payloads byte-identical (same subtime) or
|
||||
* distinct (different subtime) to exercise bug-845 dup detection. */
|
||||
static uint8_t* encode_job(uint64_t jobid, uint64_t subtime, size_t* out_len)
|
||||
{
|
||||
struct capn c;
|
||||
capn_init_malloc(&c);
|
||||
capn_ptr root = capn_root(&c);
|
||||
Job_ptr jp = new_Job(root.seg);
|
||||
struct Job cj;
|
||||
memset(&cj, 0, sizeof(cj));
|
||||
cj.jobid = jobid;
|
||||
cj.subtime = subtime;
|
||||
cj.state = 1;
|
||||
write_Job(&cj, jp);
|
||||
capn_setp(root, 0, jp.p);
|
||||
|
||||
size_t sz = capn_size(&c) + 64;
|
||||
uint8_t* buf = malloc(sz);
|
||||
int64_t n = capn_write_mem(&c, buf, sz, 0);
|
||||
capn_free(&c);
|
||||
*out_len = (size_t)n;
|
||||
return buf;
|
||||
}
|
||||
|
||||
static const char* seg_path(uint64_t start_lsn, char* out, size_t n)
|
||||
{
|
||||
snprintf(out, n, "%s/seg-%016lx.wal", g_dir, (unsigned long)start_lsn);
|
||||
return out;
|
||||
}
|
||||
|
||||
/* setUp/tearDown are owned by the shared controller_test runner, so each
|
||||
* test mints its own fresh temp WAL dir here instead. */
|
||||
static void fresh_dir(void)
|
||||
{
|
||||
strcpy(g_dir, "/tmp/xce_waldump_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(g_dir));
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* A healthy, contiguous log from LSN 1: no gaps, tip == last entry. */
|
||||
/* ------------------------------------------------------------------ */
|
||||
static void test_contiguous_log_has_no_gaps(void)
|
||||
{
|
||||
fresh_dir();
|
||||
char path[512];
|
||||
FILE* fp = fopen(seg_path(1, path, sizeof(path)), "wb");
|
||||
TEST_ASSERT_NOT_NULL(fp);
|
||||
|
||||
uint32_t last = WAL_CRC_NONE;
|
||||
put_entry(fp, 0, 1, WAL_NOOP, NULL, 0, &last, 1);
|
||||
size_t p2, p3;
|
||||
uint8_t* j2 = encode_job(1001, 100, &p2); /* distinct jobs → no dup */
|
||||
uint8_t* j3 = encode_job(1002, 100, &p3);
|
||||
put_entry(fp, 0, 2, WAL_JOB_SUBMIT, j2, (uint32_t)p2, &last, 0);
|
||||
put_entry(fp, 0, 3, WAL_JOB_SUBMIT, j3, (uint32_t)p3, &last, 0);
|
||||
free(j2);
|
||||
free(j3);
|
||||
fclose(fp);
|
||||
|
||||
waldump_stats_t st;
|
||||
int rc = waldump_path(g_dir, stdout, &st);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_EQUAL_UINT(3, st.n_entries);
|
||||
TEST_ASSERT_EQUAL_UINT(2, st.n_submits);
|
||||
TEST_ASSERT_EQUAL_UINT(0, st.n_gaps);
|
||||
TEST_ASSERT_EQUAL_UINT(0, st.n_dup_submits); /* distinct jobids */
|
||||
TEST_ASSERT_EQUAL_UINT64(1, st.first_lsn);
|
||||
TEST_ASSERT_EQUAL_UINT64(3, st.last_lsn);
|
||||
TEST_ASSERT_EQUAL_UINT64(3, st.recovered_tip);
|
||||
TEST_ASSERT_EQUAL_INT(0, st.crc_broken);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* The failover bug signature: log starts at a high LSN (empty-rejoin */
|
||||
/* caught up from the watermark) so the committed prefix is missing. */
|
||||
/* ------------------------------------------------------------------ */
|
||||
static void test_missing_prefix_flagged_as_gap(void)
|
||||
{
|
||||
fresh_dir();
|
||||
char path[512];
|
||||
/* seg file named from its first LSN, 13 (like ctrl-3 in the live repro) */
|
||||
FILE* fp = fopen(seg_path(13, path, sizeof(path)), "wb");
|
||||
TEST_ASSERT_NOT_NULL(fp);
|
||||
|
||||
uint32_t last = WAL_CRC_NONE;
|
||||
put_entry(fp, 0, 13, WAL_NOOP, NULL, 0, &last, 1);
|
||||
size_t p14, p15;
|
||||
uint8_t* j14 = encode_job(2001, 100, &p14);
|
||||
uint8_t* j15 = encode_job(2002, 100, &p15);
|
||||
put_entry(fp, 0, 14, WAL_JOB_SUBMIT, j14, (uint32_t)p14, &last, 0);
|
||||
put_entry(fp, 0, 15, WAL_JOB_SUBMIT, j15, (uint32_t)p15, &last, 0);
|
||||
free(j14);
|
||||
free(j15);
|
||||
fclose(fp);
|
||||
|
||||
waldump_stats_t st;
|
||||
int rc = waldump_path(g_dir, stdout, &st);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_EQUAL_UINT(3, st.n_entries);
|
||||
TEST_ASSERT_EQUAL_UINT64(13, st.first_lsn);
|
||||
/* prefix [1..12] absent → exactly one gap */
|
||||
TEST_ASSERT_EQUAL_UINT(1, st.n_gaps);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* bug-845: each job double-appended (same jobid at adjacent LSN). The */
|
||||
/* identical-payload case = a literal byte-for-byte double-write. */
|
||||
/* ------------------------------------------------------------------ */
|
||||
static void test_duplicate_submit_detection(void)
|
||||
{
|
||||
fresh_dir();
|
||||
char path[512];
|
||||
FILE* fp = fopen(seg_path(1, path, sizeof(path)), "wb");
|
||||
TEST_ASSERT_NOT_NULL(fp);
|
||||
|
||||
uint32_t last = WAL_CRC_NONE;
|
||||
put_entry(fp, 0, 1, WAL_NOOP, NULL, 0, &last, 1);
|
||||
/* Same jobid + same subtime → byte-identical payload (literal dup). */
|
||||
size_t pa, pb;
|
||||
uint8_t* ja = encode_job(7001, 555, &pa);
|
||||
uint8_t* jb = encode_job(7001, 555, &pb);
|
||||
put_entry(fp, 0, 2, WAL_JOB_SUBMIT, ja, (uint32_t)pa, &last, 0);
|
||||
put_entry(fp, 0, 3, WAL_JOB_SUBMIT, jb, (uint32_t)pb, &last, 0);
|
||||
/* A second job, also doubled but with a DIFFERING subtime per copy. */
|
||||
size_t pc, pd;
|
||||
uint8_t* jc = encode_job(7002, 600, &pc);
|
||||
uint8_t* jd = encode_job(7002, 601, &pd);
|
||||
put_entry(fp, 0, 4, WAL_JOB_SUBMIT, jc, (uint32_t)pc, &last, 0);
|
||||
put_entry(fp, 0, 5, WAL_JOB_SUBMIT, jd, (uint32_t)pd, &last, 0);
|
||||
free(ja); free(jb); free(jc); free(jd);
|
||||
fclose(fp);
|
||||
|
||||
waldump_stats_t st;
|
||||
int rc = waldump_path(g_dir, stdout, &st);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_EQUAL_UINT(4, st.n_submits);
|
||||
TEST_ASSERT_EQUAL_UINT(2, st.n_dup_submits); /* both pairs flagged */
|
||||
TEST_ASSERT_EQUAL_UINT(1, st.n_dup_identical); /* only the first is byte-identical */
|
||||
TEST_ASSERT_EQUAL_UINT(0, st.n_gaps);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* An interior hole (a skipped LSN between two present entries). */
|
||||
/* ------------------------------------------------------------------ */
|
||||
static void test_interior_gap_flagged(void)
|
||||
{
|
||||
fresh_dir();
|
||||
/* Two segments: [1..2] then [5..6]; 3,4 are missing. */
|
||||
char p1[512], p2[512];
|
||||
FILE* f1 = fopen(seg_path(1, p1, sizeof(p1)), "wb");
|
||||
uint32_t last = WAL_CRC_NONE;
|
||||
put_entry(f1, 0, 1, WAL_NOOP, NULL, 0, &last, 1);
|
||||
put_entry(f1, 0, 2, WAL_JOB_UPDATE, "x", 1, &last, 0);
|
||||
fclose(f1);
|
||||
|
||||
FILE* f2 = fopen(seg_path(5, p2, sizeof(p2)), "wb");
|
||||
uint32_t last2 = WAL_CRC_NONE;
|
||||
put_entry(f2, 0, 5, WAL_JOB_UPDATE, "y", 1, &last2, 1);
|
||||
put_entry(f2, 0, 6, WAL_JOB_UPDATE, "z", 1, &last2, 0);
|
||||
fclose(f2);
|
||||
|
||||
waldump_stats_t st;
|
||||
int rc = waldump_path(g_dir, stdout, &st);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_EQUAL_UINT(4, st.n_entries);
|
||||
TEST_ASSERT_EQUAL_UINT(2, st.n_segments);
|
||||
TEST_ASSERT_EQUAL_UINT64(1, st.first_lsn);
|
||||
TEST_ASSERT_EQUAL_UINT64(6, st.last_lsn);
|
||||
TEST_ASSERT_EQUAL_UINT(1, st.n_gaps); /* the 3..4 hole */
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* A CRC break truncates the recovered tip below the last physical LSN. */
|
||||
/* ------------------------------------------------------------------ */
|
||||
static void test_crc_break_sets_recovered_tip(void)
|
||||
{
|
||||
fresh_dir();
|
||||
char path[512];
|
||||
FILE* fp = fopen(seg_path(1, path, sizeof(path)), "wb");
|
||||
uint32_t last = WAL_CRC_NONE;
|
||||
put_entry(fp, 0, 1, WAL_NOOP, NULL, 0, &last, 1);
|
||||
put_entry(fp, 0, 2, WAL_JOB_UPDATE, "ok", 2, &last, 0);
|
||||
fclose(fp);
|
||||
|
||||
/* Corrupt the second entry's stored CRC in place. */
|
||||
fp = fopen(path, "r+b");
|
||||
wal_entry_hdr_t h;
|
||||
long off2 = sizeof(wal_entry_hdr_t) + 0; /* entry 1 has 0 payload */
|
||||
fseek(fp, off2, SEEK_SET);
|
||||
fread(&h, sizeof(h), 1, fp);
|
||||
h.crc ^= 0xdeadbeefu; /* break it */
|
||||
fseek(fp, off2, SEEK_SET);
|
||||
fwrite(&h, sizeof(h), 1, fp);
|
||||
fclose(fp);
|
||||
|
||||
waldump_stats_t st;
|
||||
int rc = waldump_path(g_dir, stdout, &st);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
TEST_ASSERT_EQUAL_UINT64(1, st.recovered_tip); /* stops before the bad LSN 2 */
|
||||
TEST_ASSERT_NOT_EQUAL(0, st.crc_broken);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Missing path is a hard error. */
|
||||
/* ------------------------------------------------------------------ */
|
||||
static void test_missing_path_errors(void)
|
||||
{
|
||||
waldump_stats_t st;
|
||||
int rc = waldump_path("/tmp/xce_waldump_does_not_exist_zzz", stdout, &st);
|
||||
TEST_ASSERT_LESS_THAN_INT(0, rc);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* A non-WAL file (e.g. a config blob) must ERROR, not report 0 entries.*/
|
||||
/* ------------------------------------------------------------------ */
|
||||
static void test_non_wal_file_errors(void)
|
||||
{
|
||||
fresh_dir();
|
||||
char path[512];
|
||||
snprintf(path, sizeof(path), "%s/cluster.bin", g_dir);
|
||||
FILE* fp = fopen(path, "wb");
|
||||
TEST_ASSERT_NOT_NULL(fp);
|
||||
const char junk[] = "not a wal segment — a config blob";
|
||||
fwrite(junk, sizeof(junk), 1, fp);
|
||||
fclose(fp);
|
||||
|
||||
/* Pointing at a non-seg-*.wal file must be a hard error (-EINVAL), not a
|
||||
* silent "0 entries" scan. */
|
||||
waldump_stats_t st;
|
||||
int rc = waldump_path(path, stdout, &st);
|
||||
TEST_ASSERT_LESS_THAN_INT(0, rc);
|
||||
}
|
||||
|
||||
void test_waldump(void)
|
||||
{
|
||||
RUN_TEST(test_contiguous_log_has_no_gaps);
|
||||
RUN_TEST(test_missing_prefix_flagged_as_gap);
|
||||
RUN_TEST(test_duplicate_submit_detection);
|
||||
RUN_TEST(test_interior_gap_flagged);
|
||||
RUN_TEST(test_crc_break_sets_recovered_tip);
|
||||
RUN_TEST(test_missing_path_errors);
|
||||
RUN_TEST(test_non_wal_file_errors);
|
||||
}
|
||||
444
src/controller/testsuite/waldump.c
Normal file
444
src/controller/testsuite/waldump.c
Normal file
|
|
@ -0,0 +1,444 @@
|
|||
#define _GNU_SOURCE
|
||||
#include "waldump.h"
|
||||
|
||||
#include <dirent.h>
|
||||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <inttypes.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <zlib.h>
|
||||
|
||||
#include "config.h" /* WAL_HDR_CRC_SIZE, WAL_CRC_NONE, MAX_WAL_PAYLOAD_BYTES */
|
||||
#include "controller.capnp.h" /* Job decode for WAL_JOB_SUBMIT */
|
||||
#include "wal_types.h"
|
||||
|
||||
#define WAL_SEG_FMT "%s/seg-%016" PRIx64 ".wal"
|
||||
#define MAX_SEGS 4096
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Small helpers */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
static const char* wal_type_name(uint32_t t)
|
||||
{
|
||||
switch (t) {
|
||||
case WAL_JOB_SUBMIT: return "JOB_SUBMIT";
|
||||
case WAL_JOB_START: return "JOB_START";
|
||||
case WAL_JOB_FINISH: return "JOB_FINISH";
|
||||
case WAL_JOB_FAIL: return "JOB_FAIL";
|
||||
case WAL_JOB_STOP: return "JOB_STOP";
|
||||
case WAL_JOB_RESUME: return "JOB_RESUME";
|
||||
case WAL_JOB_UPDATE: return "JOB_UPDATE";
|
||||
case WAL_JOB_RUSAGE: return "JOB_RUSAGE";
|
||||
case WAL_JOB_CKPT: return "JOB_CKPT";
|
||||
case WAL_JOB_PREASON: return "JOB_PREASON";
|
||||
case WAL_QUEUE_CTRL: return "QUEUE_CTRL";
|
||||
case WAL_WORKER_CTRL: return "WORKER_CTRL";
|
||||
case WAL_NOOP: return "NOOP";
|
||||
case WAL_CHECKPOINT: return "CHECKPOINT";
|
||||
default: return "UNKNOWN";
|
||||
}
|
||||
}
|
||||
|
||||
/* Decoded fields of a WAL_JOB_SUBMIT Cap'n Proto payload (debug view). */
|
||||
typedef struct {
|
||||
uint64_t jobid;
|
||||
uint64_t subtime; /* job_submit stamps this with time(NULL) per submit */
|
||||
uint32_t state;
|
||||
} submit_info_t;
|
||||
|
||||
/* Decode a WAL_JOB_SUBMIT payload. Mirrors restore_job() (job.c) but reads a
|
||||
* few fields only, zero-alloc. Returns 0 on success, -1 if undecodable.
|
||||
* subtime is the decisive bug-845 signal: two entries of the SAME job written
|
||||
* in one submit share a subtime (a literal double-append); two separate
|
||||
* submits (e.g. a failover retry) would differ. */
|
||||
static int decode_submit_info(const uint8_t* payload, uint32_t len,
|
||||
submit_info_t* out)
|
||||
{
|
||||
if (!payload || len == 0)
|
||||
return -1;
|
||||
|
||||
struct capn cctx;
|
||||
if (capn_init_mem(&cctx, payload, len, 0) != 0)
|
||||
return -1;
|
||||
|
||||
Job_ptr jp;
|
||||
jp.p = capn_getp(capn_root(&cctx), 0, 1);
|
||||
if (jp.p.type == CAPN_NULL) {
|
||||
capn_free(&cctx);
|
||||
return -1;
|
||||
}
|
||||
|
||||
struct Job cj;
|
||||
memset(&cj, 0, sizeof(cj));
|
||||
read_Job(&cj, jp);
|
||||
out->jobid = cj.jobid;
|
||||
out->subtime = cj.subtime;
|
||||
out->state = cj.state;
|
||||
capn_free(&cctx);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* True if `dir` directly contains at least one seg-*.wal file. */
|
||||
static int dir_has_segments(const char* dir)
|
||||
{
|
||||
DIR* d = opendir(dir);
|
||||
if (!d)
|
||||
return 0;
|
||||
int found = 0;
|
||||
struct dirent* e;
|
||||
while ((e = readdir(d)) != NULL) {
|
||||
size_t n = strlen(e->d_name);
|
||||
if (n > 4 && strncmp(e->d_name, "seg-", 4) == 0 &&
|
||||
strcmp(e->d_name + n - 4, ".wal") == 0) {
|
||||
found = 1;
|
||||
break;
|
||||
}
|
||||
}
|
||||
closedir(d);
|
||||
return found;
|
||||
}
|
||||
|
||||
/* Resolve `path` (seg file / wal dir / node-or-data dir) to the wal dir that
|
||||
* holds the seg-*.wal files. Writes it to `out`. Returns 0, or -errno. When
|
||||
* `path` is itself a single seg file, sets *single_seg to that file path. */
|
||||
static int resolve_wal_dir(const char* path, char* out, size_t outsz,
|
||||
char* single_seg, size_t segsz)
|
||||
{
|
||||
struct stat st;
|
||||
if (stat(path, &st) != 0)
|
||||
return -errno;
|
||||
|
||||
single_seg[0] = '\0';
|
||||
|
||||
if (S_ISREG(st.st_mode)) {
|
||||
/* Only a seg-*.wal file is a WAL segment. Reject anything else (e.g. a
|
||||
* config blob like conf/cluster.bin) instead of parsing it as a WAL and
|
||||
* silently reporting 0 entries. */
|
||||
const char* base = strrchr(path, '/');
|
||||
base = base ? base + 1 : path;
|
||||
size_t bl = strlen(base);
|
||||
if (strncmp(base, "seg-", 4) != 0 || bl < 5 ||
|
||||
strcmp(base + bl - 4, ".wal") != 0)
|
||||
return -EINVAL;
|
||||
snprintf(single_seg, segsz, "%s", path);
|
||||
/* wal dir = dirname(path); only used for display */
|
||||
snprintf(out, outsz, "%s", path);
|
||||
char* slash = strrchr(out, '/');
|
||||
if (slash)
|
||||
*slash = '\0';
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!S_ISDIR(st.st_mode))
|
||||
return -ENOTDIR;
|
||||
|
||||
/* A directory: try it directly, then data/wal, then wal/. */
|
||||
if (dir_has_segments(path)) {
|
||||
snprintf(out, outsz, "%s", path);
|
||||
return 0;
|
||||
}
|
||||
char cand[4096];
|
||||
snprintf(cand, sizeof(cand), "%s/data/wal", path);
|
||||
if (dir_has_segments(cand)) {
|
||||
snprintf(out, outsz, "%s", cand);
|
||||
return 0;
|
||||
}
|
||||
snprintf(cand, sizeof(cand), "%s/wal", path);
|
||||
if (dir_has_segments(cand)) {
|
||||
snprintf(out, outsz, "%s", cand);
|
||||
return 0;
|
||||
}
|
||||
return -ENOENT; /* no segments anywhere under path */
|
||||
}
|
||||
|
||||
/* Parse the start LSN out of a seg file basename ("seg-<hex16>.wal"). */
|
||||
static int parse_seg_start(const char* name, uint64_t* lsn)
|
||||
{
|
||||
return sscanf(name, "seg-%" SCNx64 ".wal", lsn) == 1 ? 0 : -1;
|
||||
}
|
||||
|
||||
/* Collect + sort segment start LSNs under wal_dir. Returns count, or -errno. */
|
||||
static int collect_segments(const char* wal_dir, uint64_t* seg_lsns, int max)
|
||||
{
|
||||
DIR* d = opendir(wal_dir);
|
||||
if (!d)
|
||||
return -errno;
|
||||
|
||||
int n = 0;
|
||||
struct dirent* e;
|
||||
while ((e = readdir(d)) != NULL && n < max) {
|
||||
uint64_t s;
|
||||
if (parse_seg_start(e->d_name, &s) == 0)
|
||||
seg_lsns[n++] = s;
|
||||
}
|
||||
closedir(d);
|
||||
|
||||
for (int i = 1; i < n; i++) { /* insertion sort — segment count is tiny */
|
||||
uint64_t key = seg_lsns[i];
|
||||
int j = i - 1;
|
||||
while (j >= 0 && seg_lsns[j] > key) {
|
||||
seg_lsns[j + 1] = seg_lsns[j];
|
||||
j--;
|
||||
}
|
||||
seg_lsns[j + 1] = key;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Per-segment scan: prints entries, validates the CRC chain, detects */
|
||||
/* LSN gaps. Carries the chain CRC and prev LSN across segment calls. */
|
||||
/* ------------------------------------------------------------------ */
|
||||
static void scan_segment(const char* seg_path, FILE* out,
|
||||
uint32_t* last_crc, uint64_t* prev_lsn,
|
||||
uint64_t* prev_submit_jobid, uint32_t* prev_submit_pcrc,
|
||||
waldump_stats_t* st)
|
||||
{
|
||||
int fd = open(seg_path, O_RDONLY);
|
||||
if (fd < 0) {
|
||||
fprintf(out, " !! cannot open %s: %s\n", seg_path, strerror(errno));
|
||||
return;
|
||||
}
|
||||
|
||||
uint32_t off = 0;
|
||||
for (;;) {
|
||||
wal_entry_hdr_t hdr;
|
||||
ssize_t hn = pread(fd, &hdr, sizeof(hdr), (off_t)off);
|
||||
if (hn == 0)
|
||||
break; /* clean EOF */
|
||||
if (hn < (ssize_t)sizeof(hdr))
|
||||
break; /* truncated / pre-allocated zeros */
|
||||
if (hdr.lsn == 0 && hdr.type == 0 && hdr.payload_len == 0)
|
||||
break; /* pre-allocated zero region = clean end */
|
||||
|
||||
/* Bound payload before trusting it (corruption guard). */
|
||||
if (hdr.payload_len > MAX_WAL_PAYLOAD_BYTES) {
|
||||
fprintf(out, " !! LSN %" PRIu64 " payload_len %u exceeds max — stop\n",
|
||||
hdr.lsn, hdr.payload_len);
|
||||
st->crc_broken = 1;
|
||||
break;
|
||||
}
|
||||
|
||||
/* CRC chain check (identical rule to persist recovery). */
|
||||
int chain_ok;
|
||||
if (off == 0)
|
||||
chain_ok = (hdr.prev_crc == WAL_CRC_NONE || *last_crc == WAL_CRC_NONE ||
|
||||
hdr.prev_crc == *last_crc);
|
||||
else
|
||||
chain_ok = (hdr.prev_crc == *last_crc);
|
||||
if (!chain_ok) {
|
||||
fprintf(out, " !! LSN %" PRIu64 " prev_crc chain break — stop\n",
|
||||
hdr.lsn);
|
||||
st->crc_broken = 1;
|
||||
break;
|
||||
}
|
||||
|
||||
/* Verify entry CRC over header + payload. */
|
||||
uint8_t* payload = NULL;
|
||||
uint32_t want_crc = crc32(0L, (const Bytef*)&hdr, WAL_HDR_CRC_SIZE);
|
||||
if (hdr.payload_len > 0) {
|
||||
payload = malloc(hdr.payload_len);
|
||||
if (!payload)
|
||||
break;
|
||||
ssize_t pn = pread(fd, payload, hdr.payload_len,
|
||||
(off_t)(off + sizeof(hdr)));
|
||||
if (pn != (ssize_t)hdr.payload_len) {
|
||||
free(payload);
|
||||
break; /* short read — clean end */
|
||||
}
|
||||
want_crc = crc32(want_crc, (const Bytef*)payload, hdr.payload_len);
|
||||
}
|
||||
if (hdr.crc != want_crc) {
|
||||
fprintf(out, " !! LSN %" PRIu64 " CRC mismatch — stop\n", hdr.lsn);
|
||||
st->crc_broken = 1;
|
||||
free(payload);
|
||||
break;
|
||||
}
|
||||
|
||||
/* Gap detection against the running LSN sequence. */
|
||||
if (*prev_lsn == 0) {
|
||||
if (hdr.lsn > 1) {
|
||||
fprintf(out, " *** GAP: missing prefix LSN 1..%" PRIu64
|
||||
" (%" PRIu64 " entries absent) ***\n",
|
||||
hdr.lsn - 1, hdr.lsn - 1);
|
||||
st->n_gaps++;
|
||||
}
|
||||
} else if (hdr.lsn > *prev_lsn + 1) {
|
||||
fprintf(out, " *** GAP: missing LSN %" PRIu64 "..%" PRIu64 " ***\n",
|
||||
*prev_lsn + 1, hdr.lsn - 1);
|
||||
st->n_gaps++;
|
||||
}
|
||||
|
||||
/* payload-only CRC — isolates the blob from the header (LSN/term/ts),
|
||||
* so two entries carrying the SAME job blob share a pcrc regardless of
|
||||
* their differing LSNs. Decisive for bug-845: equal pcrc on an adjacent
|
||||
* same-jobid pair ⇒ a literal byte-for-byte double-append. */
|
||||
uint32_t pcrc = (hdr.payload_len > 0)
|
||||
? crc32(0L, (const Bytef*)payload, hdr.payload_len)
|
||||
: 0;
|
||||
|
||||
/* Per-entry line, with jobid/subtime for submits. */
|
||||
char info[96] = "";
|
||||
int is_dup = 0;
|
||||
int dup_identical = 0;
|
||||
if (hdr.type == WAL_JOB_SUBMIT) {
|
||||
submit_info_t si;
|
||||
uint64_t jobid = 0;
|
||||
if (decode_submit_info(payload, hdr.payload_len, &si) == 0) {
|
||||
jobid = si.jobid;
|
||||
snprintf(info, sizeof(info),
|
||||
"jobid=%" PRIu64 " subtime=%" PRIu64 " state=%u",
|
||||
si.jobid, si.subtime, si.state);
|
||||
} else {
|
||||
snprintf(info, sizeof(info), "jobid=?");
|
||||
}
|
||||
st->n_submits++;
|
||||
|
||||
/* bug-845: adjacent duplicate submit — same jobid as the previous
|
||||
* entry at lsn = prev_lsn + 1. Capture identity BEFORE updating the
|
||||
* carried pcrc. */
|
||||
if (jobid != 0 && jobid == *prev_submit_jobid &&
|
||||
hdr.lsn == *prev_lsn + 1) {
|
||||
is_dup = 1;
|
||||
dup_identical = (pcrc == *prev_submit_pcrc);
|
||||
st->n_dup_submits++;
|
||||
if (dup_identical)
|
||||
st->n_dup_identical++;
|
||||
}
|
||||
*prev_submit_jobid = jobid;
|
||||
*prev_submit_pcrc = pcrc;
|
||||
} else {
|
||||
/* a non-submit entry breaks the adjacency for dup detection */
|
||||
*prev_submit_jobid = 0;
|
||||
}
|
||||
|
||||
fprintf(out, " %8" PRIu64 " | %5" PRIu64 " | %-11s | %5u | %08x | %s\n",
|
||||
hdr.lsn, hdr.term, wal_type_name(hdr.type), hdr.payload_len,
|
||||
pcrc, info);
|
||||
if (is_dup)
|
||||
fprintf(out, " ^^ DUP submit: payload %s the prior entry\n",
|
||||
dup_identical ? "BYTE-IDENTICAL to" : "DIFFERS from");
|
||||
|
||||
free(payload);
|
||||
|
||||
/* Advance watermarks. */
|
||||
if (st->first_lsn == 0 || hdr.lsn < st->first_lsn)
|
||||
st->first_lsn = hdr.lsn;
|
||||
if (hdr.lsn > st->last_lsn)
|
||||
st->last_lsn = hdr.lsn;
|
||||
if (hdr.lsn > st->recovered_tip)
|
||||
st->recovered_tip = hdr.lsn;
|
||||
st->n_entries++;
|
||||
*prev_lsn = hdr.lsn;
|
||||
*last_crc = hdr.crc;
|
||||
|
||||
off += (uint32_t)(sizeof(hdr) + hdr.payload_len);
|
||||
}
|
||||
|
||||
close(fd);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Public entry point */
|
||||
/* ------------------------------------------------------------------ */
|
||||
int waldump_path(const char* path, FILE* out, waldump_stats_t* stats)
|
||||
{
|
||||
if (!path || !out)
|
||||
return -EINVAL;
|
||||
|
||||
char wal_dir[4096];
|
||||
char single_seg[4096];
|
||||
int rc = resolve_wal_dir(path, wal_dir, sizeof(wal_dir),
|
||||
single_seg, sizeof(single_seg));
|
||||
if (rc != 0) {
|
||||
/* Clear, actionable error instead of silently scanning a non-WAL path
|
||||
* (e.g. conf/cluster.bin) and reporting "0 entries". */
|
||||
fprintf(out,
|
||||
"waldump: no WAL segments at '%s' (%s)\n"
|
||||
" point at a node's WAL dir, e.g. "
|
||||
"<deploy>/controller/ctrl-N/data/wal\n"
|
||||
" (or a specific seg-*.wal file inside it)\n",
|
||||
path, strerror(-rc));
|
||||
return rc;
|
||||
}
|
||||
|
||||
waldump_stats_t st;
|
||||
memset(&st, 0, sizeof(st));
|
||||
|
||||
uint64_t seg_lsns[MAX_SEGS];
|
||||
int nseg;
|
||||
/* wider than wal_dir[4096] so the "/seg-<hex16>.wal" suffix never truncates */
|
||||
char seg_path[4096 + 64];
|
||||
|
||||
if (single_seg[0] != '\0') {
|
||||
/* A single seg file: parse its start for ordering display. */
|
||||
const char* base = strrchr(single_seg, '/');
|
||||
base = base ? base + 1 : single_seg;
|
||||
uint64_t start = 0;
|
||||
parse_seg_start(base, &start);
|
||||
seg_lsns[0] = start;
|
||||
nseg = 1;
|
||||
snprintf(seg_path, sizeof(seg_path), "%s", single_seg);
|
||||
|
||||
/* If the dir holds OTHER segments, this single-file dump shows only a
|
||||
* slice of the log. A node that recovered then caught up keeps its
|
||||
* prefix in seg-...001.wal and the suffix in seg-...<current_lsn>.wal,
|
||||
* so a single-file dump looks (wrongly) like missing data. Steer the
|
||||
* user to the whole dir. */
|
||||
uint64_t others[MAX_SEGS];
|
||||
int total = collect_segments(wal_dir, others, MAX_SEGS);
|
||||
if (total > 1)
|
||||
fprintf(out,
|
||||
"NOTE: %s holds %d segments — this dumps only %s.\n"
|
||||
" Run 'waldump %s' (the directory) for the full "
|
||||
"stitched 1..N log.\n",
|
||||
wal_dir, total, base, wal_dir);
|
||||
} else {
|
||||
nseg = collect_segments(wal_dir, seg_lsns, MAX_SEGS);
|
||||
if (nseg < 0)
|
||||
return nseg;
|
||||
if (nseg == 0)
|
||||
return -ENOENT;
|
||||
}
|
||||
|
||||
st.n_segments = (uint32_t)nseg;
|
||||
|
||||
fprintf(out, "== WAL: %s (%d segment%s) ==\n", wal_dir, nseg,
|
||||
nseg == 1 ? "" : "s");
|
||||
fprintf(out, " %8s | %5s | %-11s | %5s | %8s | %s\n",
|
||||
"LSN", "TERM", "TYPE", "PLEN", "PCRC", "INFO");
|
||||
|
||||
uint32_t last_crc = WAL_CRC_NONE;
|
||||
uint64_t prev_lsn = 0;
|
||||
uint64_t prev_submit_jobid = 0;
|
||||
uint32_t prev_submit_pcrc = 0;
|
||||
for (int i = 0; i < nseg; i++) {
|
||||
if (single_seg[0] == '\0')
|
||||
snprintf(seg_path, sizeof(seg_path), WAL_SEG_FMT, wal_dir,
|
||||
seg_lsns[i]);
|
||||
scan_segment(seg_path, out, &last_crc, &prev_lsn,
|
||||
&prev_submit_jobid, &prev_submit_pcrc, &st);
|
||||
}
|
||||
|
||||
fprintf(out, "-- summary --\n");
|
||||
fprintf(out, " segments: %u entries: %u submits: %u\n",
|
||||
st.n_segments, st.n_entries, st.n_submits);
|
||||
fprintf(out, " lsn range: %" PRIu64 "..%" PRIu64
|
||||
" recovered tip: %" PRIu64 "\n",
|
||||
st.first_lsn, st.last_lsn, st.recovered_tip);
|
||||
fprintf(out, " gaps: %u%s crc chain: %s\n", st.n_gaps,
|
||||
(st.n_gaps && st.first_lsn > 1) ? " (committed prefix missing!)" : "",
|
||||
st.crc_broken ? "BROKEN" : "intact");
|
||||
if (st.n_dup_submits > 0)
|
||||
fprintf(out, " duplicate submits: %u (byte-identical: %u) — bug-845\n",
|
||||
st.n_dup_submits, st.n_dup_identical);
|
||||
|
||||
if (stats)
|
||||
*stats = st;
|
||||
return 0;
|
||||
}
|
||||
42
src/controller/testsuite/waldump.h
Normal file
42
src/controller/testsuite/waldump.h
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
#pragma once
|
||||
/*
|
||||
* waldump.h — read-only WAL segment inspector for failover/durability triage.
|
||||
*
|
||||
* Decodes the on-disk WAL (`wal_entry_hdr_t` + payload, see wal_types.h),
|
||||
* lists every physical entry, decodes JOB_SUBMIT jobids via the Cap'n Proto
|
||||
* Job schema, and — the decisive signal for the committed-job-loss bug —
|
||||
* flags LSN gaps (a missing prefix, or any non-contiguous hole) and reports
|
||||
* the CRC-chain "recovered tip" a real recovery would stop at.
|
||||
*
|
||||
* Strictly read-only: opens segments O_RDONLY, applies no entries, mutates no
|
||||
* persist_t state. Safe to run against a live node's data dir.
|
||||
*/
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
|
||||
/* Aggregate result of scanning one path (a seg file, a wal/ dir, or a node
|
||||
* data dir). All counts are over the physically-present, CRC-valid entries. */
|
||||
typedef struct {
|
||||
uint64_t first_lsn; /* lowest LSN seen (0 = no entries) */
|
||||
uint64_t last_lsn; /* highest LSN seen */
|
||||
uint64_t recovered_tip; /* highest LSN reachable via an intact CRC chain */
|
||||
uint32_t n_entries; /* total entries scanned */
|
||||
uint32_t n_submits; /* WAL_JOB_SUBMIT entries */
|
||||
uint32_t n_gaps; /* LSN discontinuities (missing prefix counts as 1) */
|
||||
uint32_t n_segments; /* segment files scanned */
|
||||
uint32_t n_dup_submits; /* JOB_SUBMITs with same jobid as the immediately-
|
||||
* preceding entry at an adjacent LSN (bug-845) */
|
||||
uint32_t n_dup_identical; /* of n_dup_submits, those whose payload is
|
||||
* byte-identical to the prior entry */
|
||||
int crc_broken; /* nonzero if a CRC/chain break truncated a segment */
|
||||
} waldump_stats_t;
|
||||
|
||||
/* Dump every WAL entry under `path` to `out` (human-readable) and fill
|
||||
* `*stats` when non-NULL. `path` may be:
|
||||
* - a single seg-*.wal file,
|
||||
* - a directory holding seg-*.wal files (a wal/ dir), or
|
||||
* - a node/data dir whose wal/ subdir (data/wal or wal) holds the segments.
|
||||
* Returns 0 on success, or a negative errno on a hard I/O error (e.g. the
|
||||
* path does not exist / no segments found). Gaps and CRC breaks are reported
|
||||
* through `out`/`stats`, NOT via the return code. */
|
||||
int waldump_path(const char* path, FILE* out, waldump_stats_t* stats);
|
||||
|
|
@ -90,6 +90,70 @@ static void test_wal_batch_roundtrip_payload_bytes(void)
|
|||
printf("test_wal_batch_roundtrip_payload_bytes: PASS\n");
|
||||
}
|
||||
|
||||
/* Production-scale batch: many entries each carrying a full-size (872-byte)
|
||||
* capnp-Job-sized payload — mirrors the live catch-up stream where ctrl-1
|
||||
* rejoined and received header-only entries (PLEN=0). The small 2-entry
|
||||
* round-trip passes; this reproduces whatever the larger/multi-segment batch
|
||||
* does on the wire (bug-847). */
|
||||
static void test_wal_batch_roundtrip_large_payloads(void)
|
||||
{
|
||||
/* KNOWN-FAILING REPRODUCER for bug-847: a production-scale batch that spans
|
||||
* more than one capnp segment loses its per-entry payload Data pointers on
|
||||
* the wire (decode returns raw_payload_len=0; it also leaks the orphaned
|
||||
* decode allocations under ASAN). IGNORE'd so the suite stays green; REMOVE
|
||||
* this TEST_IGNORE line when bug-847 is fixed and assert the payloads
|
||||
* survive. Decisive evidence: this fails "Expected 872 Was 0" while the
|
||||
* 2-entry round-trip above passes. */
|
||||
TEST_IGNORE_MESSAGE(
|
||||
"bug-847: catch-up batch drops payloads across capnp segments — "
|
||||
"un-ignore when fixed");
|
||||
|
||||
enum { N = 14, PSZ = 872 };
|
||||
wal_catchup_entry_t e[N];
|
||||
wal_catchup_entry_t *entries[N];
|
||||
uint8_t* payloads[N];
|
||||
|
||||
for (int i = 0; i < N; i++) {
|
||||
payloads[i] = malloc(PSZ);
|
||||
TEST_ASSERT_NOT_NULL(payloads[i]);
|
||||
/* distinct, non-zero content per entry so a dropped/zeroed payload is
|
||||
* unambiguous */
|
||||
for (int b = 0; b < PSZ; b++)
|
||||
payloads[i][b] = (uint8_t)((i * 31 + b) & 0xff);
|
||||
|
||||
memset(&e[i], 0, sizeof(e[i]));
|
||||
e[i].lsn = (uint64_t)(i + 1);
|
||||
e[i].entry_type = 1; /* WAL_JOB_SUBMIT */
|
||||
e[i].raw_payload = payloads[i];
|
||||
e[i].raw_payload_len = PSZ;
|
||||
entries[i] = &e[i];
|
||||
}
|
||||
|
||||
io_packet_t* pkt = encode_batch(entries, N);
|
||||
xce_request_t req;
|
||||
memset(&req, 0, sizeof(req));
|
||||
TEST_ASSERT_EQUAL_INT(0, xce_decode(pkt, &req));
|
||||
|
||||
wal_catchup_entry_t* batch = req.body.walcatchupentrymsg;
|
||||
TEST_ASSERT_NOT_NULL(batch);
|
||||
TEST_ASSERT_EQUAL_INT(N, batch->n_entries);
|
||||
|
||||
for (int i = 0; i < N; i++) {
|
||||
TEST_ASSERT_NOT_NULL(batch->entries[i]);
|
||||
TEST_ASSERT_EQUAL_UINT64((uint64_t)(i + 1), batch->entries[i]->lsn);
|
||||
/* the bug: these come back NULL / len 0 on the live path */
|
||||
TEST_ASSERT_EQUAL_size_t(PSZ, batch->entries[i]->raw_payload_len);
|
||||
TEST_ASSERT_NOT_NULL(batch->entries[i]->raw_payload);
|
||||
TEST_ASSERT_EQUAL_MEMORY(payloads[i], batch->entries[i]->raw_payload,
|
||||
PSZ);
|
||||
}
|
||||
|
||||
xce_free_request(&req);
|
||||
for (int i = 0; i < N; i++)
|
||||
free(payloads[i]);
|
||||
printf("test_wal_batch_roundtrip_large_payloads: PASS\n");
|
||||
}
|
||||
|
||||
static void test_wal_batch_entry_without_payload(void)
|
||||
{
|
||||
/* An entry with no payload (e.g. header-only types) must decode with
|
||||
|
|
@ -215,6 +279,7 @@ static void test_snapshot_chunk_empty_payload(void)
|
|||
void test_wal_wire(void)
|
||||
{
|
||||
RUN_TEST(test_wal_batch_roundtrip_payload_bytes);
|
||||
RUN_TEST(test_wal_batch_roundtrip_large_payloads);
|
||||
RUN_TEST(test_wal_batch_entry_without_payload);
|
||||
RUN_TEST(test_wal_batch_encode_rejects_invalid_args);
|
||||
RUN_TEST(test_snapshot_chunk_roundtrip);
|
||||
|
|
|
|||
51
uninstall-code-review-graph.sh
Executable file
51
uninstall-code-review-graph.sh
Executable file
|
|
@ -0,0 +1,51 @@
|
|||
#!/usr/bin/env bash
|
||||
# Fully uninstall `code-review-graph` (the Python/uv MCP tool).
|
||||
#
|
||||
# IMPORTANT: Run this ONLY after closing every Claude Code session that has
|
||||
# this worktree registered as an MCP server. While a session is live it
|
||||
# re-spawns `uvx code-review-graph serve`, which self-bootstraps: it
|
||||
# reinstalls the uv package and rewrites .mcp.json / .claude/settings.json
|
||||
# hooks within ~10s. The configs are already cleaned on disk; this script
|
||||
# removes the package + binaries + leftover state once nothing is respawning it.
|
||||
#
|
||||
# This does NOT touch `codegraph` (the separate @colbymchenry/codegraph bun tool).
|
||||
set -uo pipefail
|
||||
|
||||
WT="/home/rshen/projects/xce/worktrees/persistent_ha"
|
||||
UVENV="/home/claude/tools/uv"
|
||||
|
||||
echo "==> Killing any running code-review-graph / crg-daemon processes"
|
||||
pkill -f "code-review-graph serve" 2>/dev/null || true
|
||||
pkill -f "code-review-graph.*daemon" 2>/dev/null || true
|
||||
pkill -f "uvx code-review-graph" 2>/dev/null || true
|
||||
pkill -f "crg-daemon" 2>/dev/null || true
|
||||
sleep 1
|
||||
|
||||
echo "==> Uninstalling uv package"
|
||||
VIRTUAL_ENV="$UVENV" uv pip uninstall code-review-graph 2>/dev/null || true
|
||||
|
||||
echo "==> Removing leftover binaries"
|
||||
rm -f "$UVENV/bin/code-review-graph" "$UVENV/bin/crg-daemon"
|
||||
|
||||
echo "==> Removing state directories"
|
||||
rm -rf "$WT/.code-review-graph" \
|
||||
/home/rshen/projects/xce/xce/.code-review-graph \
|
||||
/home/claude/.code-review-graph \
|
||||
"/home/claude/.cache/claude-cli-nodejs/-home-rshen-projects-xce-worktrees-persistent-ha/mcp-logs-code-review-graph"
|
||||
|
||||
echo "==> Removing git pre-commit hook (if it is the code-review-graph one)"
|
||||
HOOK="/home/rshen/projects/xce/xce/.git/hooks/pre-commit"
|
||||
if [ -f "$HOOK" ] && grep -q "code-review-graph" "$HOOK"; then rm -f "$HOOK"; fi
|
||||
|
||||
echo "==> Purging uv cache entries"
|
||||
find /home/claude/.cache/uv -maxdepth 4 -iname "*code_review_graph*" -exec rm -rf {} + 2>/dev/null || true
|
||||
|
||||
echo "==> Verification"
|
||||
if command -v code-review-graph >/dev/null 2>&1; then
|
||||
echo " STILL PRESENT — a Claude session is probably still respawning it. Close it and re-run."
|
||||
else
|
||||
echo " code-review-graph: GONE"
|
||||
fi
|
||||
echo " .mcp.json refs: $(grep -c code-review-graph "$WT/.mcp.json" 2>/dev/null || echo 0)"
|
||||
echo " settings.json hook refs: $(grep -c code-review-graph "$WT/.claude/settings.json" 2>/dev/null || echo 0)"
|
||||
echo "Done."
|
||||
Loading…
Add table
Add a link
Reference in a new issue