wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

docs(specs): gap-analysis validation pass across all 9 HA-family specs

Browse source 
Appended "Gap Analysis — Validation Pass (2026-06-09)" to each spec's
research.md with requirement-to-asset maps, status tags, and effort/risk.

Key outcomes:
- ha-repl-hardening: REOPENED with task series 7.1-7.7 — replication
  payload data plane is missing (snapshot chunks and WAL catch-up ship
  header-only; standbys converge on LSN without holding entity state)
- worker-join-refactor, resval-id-decoupling: marked implemented
  (all criteria re-verified, full test suite green)
- buglog: 335/336 (scheduler dequeue, open), 337/338/339 (fixed in
  previous commit); cerebrum Decision Log corrected (term-only vote
  deny was recorded as intentional but was a regression)
This commit is contained in:
Rongsong Shen 2026-06-10 01:26:08 +00:00
parent 9821ec4f3c
commit cdd446fc17
21 changed files with 3955 additions and 288 deletions
Download patch file Download diff file Expand all files Collapse all files

106
.kiro/specs/election-iomgr-refactor/research.md
View file

@ -273,3 +273,109 @@ The `election-iomgr-refactor` spec is **fully implemented and verified**. No blo
## Status: CLOSED — No remaining gaps
All 15 implementation tasks (1.1–5.3) and all 5 validation tasks (6.1–6.5) confirmed done. The spec is in `implemented` phase. Deferred cleanup items (dead code: `am_i_primary`, `ha_wait_for_heartbeat_or_timeout`, `start_heartbeat_listener`/`stop_heartbeat_listener` stubs) are cosmetic and tracked as tech debt.
---
# Gap Analysis — Validation Pass (2026-06-09)
> Spec phase: `implemented`. This pass audits requirements-vs-CURRENT-codebase drift,
> analyzing the working tree **as it is on disk** — including uncommitted changes to
> `ha.c`, `ha.h`, `myapp.c`, and `src/lib/raft/raft.c` (post `ha-repl-hardening` /
> "finalize configuration replication" work).
## Analysis Summary
- **Implementation status**: 8 of 9 requirements **Implemented** in the current tree; Req 9 (no regression) is **Partial** — the working tree has **2 failing controller unit tests** introduced by the uncommitted `ha.c` vote-path changes (verified by building and running `controller_unit_test`).
- **Architecture drift since the spec was written (benign)**: the election core was extracted into a reusable raft library (`src/lib/raft/raft.c`), promotion now uses heartbeat-reply **majority convergence** (`ha_check_majority_convergence`) as the primary mechanism with `ha_repl_ack_received` retained, the committing timeout default moved 10s → 60s, and raft init moved from per-election (`ha_start_election_async`) to one-time startup (`ha_init_raft`, uncommitted). All of these still satisfy the requirement text.
- **Real gaps**: (1) two unit tests contradict the new vote-deny semantics; (2) semantic divergence between the legacy `handle_request_vote` path (ha.c) and `raft_on_vote_request` (raft.c) on duplicate-vote handling; (3) the new `myapp_run` skip-election "self" sentinel does not match production self-vote naming; (4) stale comments + known dead code.
- **Effort to close**: S (≤1 day). **Risk**: Medium (election-safety semantics must be decided deliberately, not just patched to make tests green).
## Requirement-to-Asset Map
| Req | Summary | Current Assets (on-disk) | Status |
|-----|---------|--------------------------|--------|
| 1.1–1.3 | No blocking vote pthreads; vote RPC via standard I/O; probe timeout = deny | `ha_start_election_async` (ha.c:3030) delegates to `raft_start_election`; `vote_request_thread`/`am_i_primary` deleted (tombstones ha.c:885–887, ha.h:211); timeout via `raft_cb_schedule_timeout` → `delay_task_ms` | Implemented |
| 2.1–2.4 | Async TCP+TLS via iomgr; TLS client ctx; send vote after handshake; connect failure = deny | `endpoint_set_on_connected` (netio.c:782, fired at netio.c:441–443 with fire-once semantics); `ha->client_ssl_ctx` applied in vote-ep setup (ha.c:160–167); `on_vote_connected` (ha.c:2291); `vote_ep_err_handler` (ha.c:2945+) | Implemented |
| 3.1–3.4 | Reply via iomgr dispatch; no capnp UAF; field extraction; term fencing | `vote_reply_msg_handler` + role-routing `peer_ctrl_msg_handler` (ha.c:2565–2579); fields read before free; term fencing in `raft_on_vote_reply` (raft.c) | Implemented |
| 4.1 | Quorum → CTRL_PRIMARY_COMMITTING | `raft_cb_become_leader` → role store (ha.c:505–507) | Implemented |
| 4.2–4.3 | Freshness gate = ground truth; cold start only highest config_tag wins | `raft_cb_vote_eligible` app-policy gate invoked from cluster.c:331–373 (`raft_on_vote_request` step 5); tag=0 node refuses to campaign (ha.c:3133–3145) | Implemented |
| 4.4–4.5, 4.7 | Split-vote/no-reply retry; retry indefinitely | `raft_on_election_timeout` (raft.c:250), `raft_on_leader_lost` (raft.c:268) with priority-scaled backoff; heartbeat watchdog re-arms (myapp.c skip-election path now arms `ha_arm_heartbeat_timeout_vtask` — uncommitted fix) | Implemented |
| 4.6 | ELECTING + heartbeat from new PRIMARY → STANDBY | Achieved via an **equivalent mechanism**, not literal heartbeat: `ha_dispatch_gate` denies heartbeat/ctrlhello in CTRL_ELECTING; the candidate instead becomes STANDBY via vote-grant → `raft_cb_become_follower` (ha.c:596–659), and the winner's ctrlhello is retried each heartbeat tick once the loser is STANDBY | Implemented (mechanism drift vs. requirement wording — document, don't change) |
| 5.1–5.4 | Dispatch-gate per role | `ha_dispatch_gate` (ha.c:1412): ELECTING allows requestVote+probeConfigTag only; COMMITTING allowlist (ha.c:1436–1454); STANDBY allowlist; PRIMARY full. Note: CTRL_INIT now also admits OPCode_heartbeat (deliberate extension for INIT→STANDBY) | Implemented |
| 5.5 | Monotonic persisted term | `ha_persist_term`/`ha_restore_term` (ha.c:1647/1694); `raft_cb_persist` (ha.c:677) persists before grant; `ha_init_raft` (uncommitted, ha.c:2984) seeds raft from restored term **once** at startup — fixes term reset across re-elections | Implemented |
| 6.1 | PRIMARY/COMMITTING deny all votes | raft.c role gate (step 3, equal-term challenger) + ha.c:1838 legacy gate + term-ordered demote-before-grant (ha.c:1805–1825) | Implemented |
| 6.2 | STANDBY + live primary denies votes | **Drift**: ha.c:1840 (uncommitted) now requires `primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha)`. This matches the requirement text ("live primary **connection**") *more* literally than before, but `test_standby_fresh_heartbeat_denies_vote_request` (test_ha_vote_gate.c) was not updated and **FAILS** (fd defaults to −1 in `ha_init`) | Partial — code OK per requirement, test broken |
| 6.3, 6.5 | HB timeout → ELECTING → new primary | heartbeat watchdog → `ha_on_primary_ctrl_fd_rdhup` (ha.c:3183) → `ha_start_election_async`; verified by `test_ha_rdhup_integration.c` (still registered, meson.build:120) | Implemented |
| 6.4 | INIT/ELECTING evaluate term/voted_for/freshness gates | Production path: `raft_on_vote_request` (raft.c:106+). **Drift**: uncommitted raft.c change adds **idempotent re-grant** to the *same* candidate (raft.c step 4), while uncommitted ha.c change makes the legacy `handle_request_vote` **candidate-agnostic deny** (any second vote in term denied, ha.c:1851). The two vote paths now disagree; `test_vote_granted_to_same_candidate_same_term` (test_ha_election.c:432) **FAILS** | Partial — semantic divergence + failing test |
| 7.1–7.3 | Winner pushes config + WAL; tag=0 standby never fetches | `push_config_to_standbys` + `ha_send_ctrlhello_to_all_standbys` (cluster.c:1376) in COMMITTING entry (ha.c:544–559); heartbeat-driven repl decision loop (HA_REPL_PUSH_CONFIG / WAL / snapshot, ha.c:1288+); `fetch_config_from_primary` removed from production (myapp.c:289) | Implemented |
| 7.4 | Per-standby ack before counting toward majority | Evolved beyond the spec: per-standby `standby_converged[]` tracked from heartbeat `stateReport` replies; plus `OPCode_replCatchUpDone` → `ha_repl_ack_received` (ha.c:2485) | Implemented |
| 8.1–8.3 | COMMITTING blocks jobs; majority ack → PRIMARY | gate case (ha.c:1436); `ha_check_majority_convergence` (ha.c:1244, primary mechanism) and `ha_repl_ack_received` (ha.c:2923, quorum −1 math) → `ha_promote_to_primary` | Implemented |
| 8.4 | Configurable ack timeout → degraded open | `committing_timeout_fn` (ha.c:2875+) with converged/pending WARN detail; `HA_DEFAULT_COMMITTING_TIMEOUT_MS = 60000` (ha.h:131) — design said 10000, requirement only demands "configurable" | Implemented (design-doc drift only) |
| 9.1 | Single-node self-elect, no peer contact | `raft_start_election` n_peers==1 shortcircuit (raft.c:83); ctx freed inline (ha.c:3152–3157); zero-peer guard (ha.c:1981) | Implemented |
| 9.2 | Winner never fetches config | Freshness gate + push model; no fetch call sites remain (stale comment at ha.c:3136 still mentions the removed `fetch_config_from_primary`) | Implemented |
| 9.3 | RDHUP → re-election; promotion is hooks-only | `ha_on_primary_ctrl_fd_rdhup` + `phase2_done` skip; `ha_promote_to_primary` | Implemented |
| 9.4 | Priority stagger | `election_priority_step_ms` stagger in myapp.c:225–231 and watchdog offset ha.c:1597; raft `my_priority` backoff (raft.c:273) | Implemented |
| 9.* | "All existing behaviors preserved" | **2 unit tests fail in the working tree** (see below); everything else green in the run performed for this audit | **Partial** |
### Verified test failures (working tree, `controller_unit_test`, ASAN build)
```
test_vote_granted_to_same_candidate_same_term:FAIL: Expected 1 Was 0
test_standby_fresh_heartbeat_denies_vote_request:FAIL: Expected 0 Was 1 (Req 6.2)
```
Both are caused by the uncommitted `handle_request_vote` changes in ha.c:
1. ha.c:1840 — deny now additionally requires `primary_ctrl_fd >= 0` (test sets fresh heartbeat but fd = −1 from `ha_init`).
2. ha.c:1851 — duplicate-vote deny is now term-scoped and candidate-agnostic (denies same-candidate retransmission), whereas raft.c (the production vote path via cluster.c:373) deliberately keeps the idempotent same-candidate re-grant.
### Additional drift findings (no test failure yet)
- **`myapp_run` skip-election "self" sentinel is ineffective in production** (myapp.c, uncommitted): the check `strcmp(voted_for_name, "self") != 0` assumes self-votes are recorded as `"self"`, but `raft_cb_persist` (ha.c:700–713) stores the node's **real candidate name** on a self-vote; `"self"` is only the test-mode/out-of-range placeholder. A restarted node that voted for itself would be classified "leader known" and skip its own election. Mitigated by the newly-armed heartbeat watchdog (it will fire and trigger a fresh election after `heartbeat_timeout_ms`), so the cost is delayed availability, not a stuck node — but the sentinel should also compare against `c->myconf->name`.
- **Stale comments**: ha.c ~826–849 still describe per-election raft re-init ("overwritten by `ha_raft_reinit()` called from `ha_start_election_async`") which the uncommitted `ha_init_raft` refactor removed; ha.c:3136 references the deleted `fetch_config_from_primary`.
- **No test coverage for `ha_init_raft`**: the new one-time-init invariant (raft state must survive across election cycles; double-init would reset term/voted_for) has no dedicated unit test; no testsuite file references `ha_init_raft`.
- **Known dead code still compiled** (carried from prior audits): `ha_wait_for_heartbeat_or_timeout` (ha.c:1587, ha.h:551).
## Implementation Approach Options
Since the spec is implemented, options address only the remaining gap closure.
### Option A: Reconcile in place (extend existing) — Recommended
- Decide the intended duplicate-vote semantics **once** and align all three artifacts:
- Standard Raft (and raft.tla referenced in `src/controller/CLAUDE.md`) permits idempotent re-grant to the same candidate; raft.c already implements this. Recommended: restore the `strcmp(..., candidate_id) != 0` condition in `handle_request_vote` (ha.c:1851) so the legacy/test path matches raft.c, keeping `test_vote_granted_to_same_candidate_same_term` valid.
- Update `test_standby_fresh_heartbeat_denies_vote_request` to set `ha.primary_ctrl_fd = <valid fd>` (the code change matches Req 6.2's "live primary connection" wording and should stay); add a sibling case asserting STANDBY + fresh heartbeat + fd = −1 → **grant** (no live connection).
- Fix the myapp.c sentinel: treat `voted_for_name == c->myconf->name` the same as `"self"`.
- Clean stale comments (ha.c:826–849, ha.c:3136); optionally delete `ha_wait_for_heartbeat_or_timeout`.
- Add a small `ha_init_raft` unit test (init once, run two election cycles, assert term monotonic / voted_for survives).
- Trade-offs: ✅ smallest diff, keeps both vote paths consistent; ❌ keeps the duplicated legacy vote path alive.
### Option B: Retire the legacy vote path (new/removed components)
- Delete `handle_request_vote` from ha.c entirely; port its tests to drive `raft_on_vote_request` + `raft_cb_vote_eligible` (the actual production gate via cluster.c). Single source of truth for vote semantics.
- Trade-offs: ✅ eliminates the dual-path divergence class permanently; ❌ larger test-port effort (test_ha_election.c, test_ha_vote_gate.c re-plumb), M-size, riskier right before closing the hardening branch.
### Option C: Hybrid
- Apply Option A now (unblock the test suite and the branch), file the Option B consolidation as a cleanup spec item alongside the existing dead-code list.
- Trade-offs: ✅ immediate green suite + tracked debt; ❌ divergence risk persists until the cleanup lands.
## Effort and Risk
| Gap | Effort | Risk | Justification |
|-----|--------|------|---------------|
| Reconcile duplicate-vote semantics (ha.c ↔ raft.c) + fix test | S | Medium | One-line condition either way, but it is election-safety semantics — verify against raft.tla checklist before choosing |
| Update Req 6.2 test for `primary_ctrl_fd` gate (+ new fd=−1 grant case) | S | Low | Code already matches requirement text; test-only change |
| myapp.c "self" sentinel fix | S | Medium | Touches startup skip-election logic; needs a 3-node restart scenario to validate (deploy-ha) |
| `ha_init_raft` unit test | S | Low | Established Unity patterns in test_ha_election.c |
| Stale comments + dead-code removal | S | Low | Cosmetic; callers already verified absent |
| Option B legacy-path retirement (if chosen) | M | Medium | Test re-plumb across 2 suites; no production behavior change intended |
## Recommendations / Next Steps
1. **Do not close the branch with the suite red.** Fix the two failing tests via Option A (preferred) before committing the working-tree changes; the current diff would fail `/xce-build controller`.
2. **Preferred approach: Option C** — Option A reconciliation now, Option B (single vote path) queued with the existing dead-code cleanup list.
3. **Keep**: `ha_init_raft` one-time init, the `primary_ctrl_fd >= 0` deny refinement, the term-scoped `voted_for` check in myapp.c, and the skip-election watchdog arming — all are correctness improvements consistent with the requirements.
4. **Requirement-text note**: Req 4.6's "heartbeat received while ELECTING → STANDBY" is satisfied by an equivalent mechanism (vote-grant → follower + primary ctrlhello retry). Record this as accepted drift rather than reworking the dispatch gate.
## Research Needed
- **Duplicate-vote semantics decision**: confirm against `raft/raft.tla` (Election Safety invariant) that same-candidate re-grant is safe in XCE's persistence model (vote persisted before grant → re-grant is replay-safe). Expected answer: yes; then align ha.c to raft.c.
- **"self" sentinel scenario**: reproduce with deploy-ha — node self-votes (candidate), is killed mid-election, restarts within the same term; verify it does not skip its own election indefinitely.
- **Full regression + ASAN pass**: run `/xce-build controller` after fixes; this audit ran only `controller_unit_test` (2 failures, no ASAN errors observed in that run).

211
.kiro/specs/ha-config-bootstrap/research.md
View file

@ -499,3 +499,214 @@ No new code needed. Proceed directly to manual integration test execution:
3. Record results in the Results Template at the bottom of `manual-tests.md`.
4. Mark tasks 11.1–11.5 `[x]` on pass.
5. Run `nix build .#xcecore_debug` for 11.5 ASAN verification.
---
# Gap Analysis — Validation Pass (2026-06-09)
## Scope and Method
Validation pass against the working tree on branch `persistent_ha` (including uncommitted changes to `ha.c`, `ha.h`, `myapp.c`, `raft.c`). All 50 acceptance criteria across the 10 requirements were checked against on-disk code. Note that the architecture has evolved past this spec's original design via four successor specs (`raft-library`, `election-iomgr-refactor`, `ha-heartbeat-liveness`, `ha-repl-hardening`, `ha-peer-connection-reuse`): the blocking election (`am_i_primary_quorum` + semaphore wait) was replaced by a fully async raft-library-driven election inside iomgr, and the standby-pull config bootstrap was replaced by a primary-push model. Several "gaps" below are therefore *sanctioned deviations*, not defects — they need requirements-doc reconciliation, not code.
All checkboxes in `tasks.md` (1.1–11.5) are marked `[x]`, including 9.3+, and manual integration tests 11.1–11.4 are recorded as executed 2026-06-03. The memory note "9.3+ remaining" is stale.
## 1. Requirement-to-Asset Map
Status legend: **Implemented** / **Partial** / **Missing** / **Unknown** / **Constraint** (sanctioned architectural deviation needing spec reconciliation).
### Requirement 1 — Listener and Connection Framework Before Election
| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 1.1 listener bound before election | Implemented | `myapp.c:113` `cluster_open_listener()` before `ha_start_election_async()`; `cluster_register_listener` at `myapp.c:138` |
| 1.2 iomgr/vtaskmgr before election | Implemented | `myapp.c:122-131` iomgr+vtaskmgr init/start precede election (Step 3/4) |
| 1.3 not-ready rejection in INIT/ELECTING | Implemented | `ha_dispatch_gate()` `ha.c:1412` returns `-XCE_ERR_NOT_READY`; wired at vtask dispatch entry `cluster.c:226` |
| 1.4 standby error includes redirect host/port | **Partial** | `XCE_ERR_STANDBY` / `E_XCE_REDIRECT` returned (`ha.c:1481`, `join.c:241`), `ha_should_redirect()` exists, capnp fields `redirectHost/redirectPort` exist (`controller.capnp:565-566`) — **but no reply path ever populates `redirect_host/redirect_port`**; replies are sent with `data_which = nodata` |
### Requirement 2 — Quorum-Based Controller Election
| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 2.1 strict-majority quorum | Implemented | `raft.c:223-234` quorum = `n_peers/2 + 1`; `ha_elect_evaluate_and_signal` `ha.c:1995` |
| 2.2 lower-term deny | Implemented | `raft_on_vote_request` Step 1 (`raft.c:110`) |
| 2.3 higher-term update + clear vote | Implemented | `raft.c:121-130` (also steps a live leader down to follower) |
| 2.4 freshness gate deny | Implemented | `raft_cb_vote_eligible` `ha.c:202`; `handle_request_vote` `ha.c:1791` |
| 2.5 catch-up standby vote denied | Implemented | `in_catch_up` check `ha.c:218`, `ha.c:1845` |
| 2.6 one vote per term | Implemented | `raft.c:157-164` (with idempotent re-grant to same candidate — safe extension) |
| 2.7 yield to fresher peer | Implemented | yield gate `ha.c:2017-2031` (`max_resp_tag/max_resp_lsn`) |
| 2.8 tag-zero candidate vs configured peer → fatal exit | **Constraint** | Implemented as *voter-only standby*, not fatal exit: `ha.c:3133-3145` (tag==0 && n_peers>1 → CTRL_STANDBY, wait for push). Strictly safer for ops (node self-heals via config push) but deviates from the EARS text "exit with a fatal error" |
| 2.9 single-node immediate self-elect | Implemented | `raft.c:83-88` `RAFT_ELECTED_IMMEDIATELY`; zero-peer guard `ha.c:1985` |
| 2.10/2.11 split vote → one retry w/ jitter+term++, then standby | **Constraint** | raft library becomes FOLLOWER on no-quorum/timeout (`raft.c:237-244`, `raft_on_election_timeout`); retries are driven indefinitely by the standby liveness watchdog → `raft_on_leader_lost` → delayed `raft_check_and_start_election` (`raft.c:268-290`). Net behavior: unbounded jittered retry loop instead of exactly-once retry. Liveness outcome equivalent or better; spec text outdated |
| 2.12 identical-freshness tiebreak by jitter, not priority | **Partial** | Winner choice never uses priority values; but election *timing* uses deterministic priority-based stagger (`myapp.c:229-239` `priority * election_priority_step_ms`; `raft.c:272-273` `priority * election_timeout_ms`). Priority is repurposed as stagger index, not vote weight — spec wording should be reconciled |
### Requirement 3 — Config Bootstrap for Unconfigured Nodes
| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 3.1 probe all peers' config tag before candidacy | **Missing (as specified)** | `xce_probe_config_tag()` fully implemented (`config_repl.c:1140`, single-RTT SSL probe) and unit-tested, but **never called in production** — `bootstrap_config_if_needed()` was removed from `myapp_run` (see Constraints in prior pass). Replacement: unconditional voter-only gate when `my_config_tag==0 && n_peers>1` (`ha.c:3133`) |
| 3.2 voter-only participation, deferred fetch | Implemented (pivoted) | voter-only path `ha.c:3133-3145`; config arrives via primary push (`push_config_to_standbys` from `raft_cb_become_leader`/COMMITTING driver) |
| 3.3 all peers tag-zero → normal candidacy | **Partial / Unknown** | No probe means a tag-0 node cannot distinguish "peers have config" from "all-zero cold start"; it always goes voter-only first. Recovery relies on the heartbeat watchdog firing → `ha_on_primary_ctrl_fd_rdhup` → `raft_on_leader_lost` re-election, which **bypasses** the tag-0 gate (`ha.c:3183-3291`). All-zero cold start thus converges, but only after `heartbeat_timeout_ms` (~10s) delay per round; not covered by an automated test |
| 3.4 exponential-backoff probe retry 500ms→10s | **Missing (as specified)** | No probe retry loop exists; the passive watchdog loop replaces it. Spec text obsolete under the push model |
| 3.5 deferred fetch from proven winner | Implemented (pivoted) | Primary pushes config + ctrlhello to all standbys on COMMITTING (`ha.c:2087-2103`); `fetch_config_from_primary`/`do_one_config_fetch` standby-pull removed by design (CLAUDE.md: standbys never dial out) |
### Requirement 4 — Startup Validation and Safety Checks
| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 4.1 even-N peer rejection | Implemented | `validate_peer_count()` `startup_validation.c:90`; double-enforced in `raft_init` (`raft.c:21`) |
| 4.2 cert/key pre-flight with full path | Implemented | `validate_cert_files()` `startup_validation.c:106` (-ENOENT, path in message), wired in `main.c` |
| 4.3 vcpus >= 1 | Implemented | `validate_vcpus()` `startup_validation.c:25`; `--vcpus` argp option in `main.c` |
| 4.4 config-file vcpus deprecation WARN | Implemented | `validate_startup_config()` `startup_validation.c:46`; warned at `myapp.c:80-86` |
### Requirement 5 — Majority-Commit WAL Replication
| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 5.1 job submit / state transitions / queue changes majority-committed | **Partial** | Only `WAL_JOB_SUBMIT` routes through `persist_wal_sync_append()` (`persist.c:1118`). Job state transitions (`WAL_JOB_START/FINISH/FAIL/...`, `persist.c:1163`) and queue changes (`WAL_QUEUE_CTRL`, `persist.c:1194`) use `persist_wal_async()` — no majority wait |
| 5.2 host state / checkpoint refs async | Implemented | worker/host events → `persist_wal_async`; checkpoint markers never replicated (`persist.c:395`) |
| 5.3 standby durable write + ACK with LSN | Implemented | `OPCode_walAck` flow, `repl_on_ack` (`repl.c:178`), `acked_lsn` tracking |
| 5.4 lag eviction + catch-up trigger | Implemented | `pending_ack_count > max` eviction with inline catch-up start, `repl.c:214-222` |
| 5.5 commit timeout → error to caller | **Constraint** | Deliberate *degraded mode*: `persist_wal_sync_append` reports `-ETIMEDOUT/-ENODEV` via `sync_rc_out` but returns 0 and the operation proceeds with local-only durability (`persist.c:1003-1043`; hook comment `persist.c:1114`). This is an AP-leaning deviation from the spec's CP statement — documented in code, not in requirements |
| 5.6 truncate above committed LSN on promotion | Implemented | `ha_compute_safe_committed_lsn()` from voter replies (`ha.c:373-437`) → `persist_truncate_wal_above(safe_lsn)` in `raft_cb_become_leader` (`ha.c:471`); per controller CLAUDE.md this is the verified-against-TLA+ design |
| 5.7 truncate on catch-up reconnect | **Partial** | RDHUP/watchdog failover path truncates to `last_known_committed_lsn` before re-election (`ha.c:3204-3207`). A standby that *restarts* (rather than detecting RDHUP) replays its full WAL including uncommitted tail; no truncation occurs before it rejoins and receives catch-up entries — divergent-tail handling on the restart path is unverified |
| 5.8 single-node local write satisfies majority | **Partial** | `repl_wait_majority_ack` returns 0 when `n_active == 0` (`repl.c:384`), but `persist_wal_sync_append` short-circuits *before* calling it when `nstandbys == 0` and sets `sync_err = -ENODEV` **without advancing `committed_lsn`** (`persist.c:1023-1031`). Single-node `committed_lsn` appears to advance only via the election-time safe-LSN path — runtime advancement per entry is not evident. Needs verification (interacts with WAL trim guard, which uses `committed_lsn` as threshold) |
### Requirement 6 — Persistent Standby-to-Primary Control Channel
| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 6.1 standby establishes persistent TLS connection to primary | **Constraint** | Direction inverted by deliberate design (controller CLAUDE.md "Inbound vs. Outbound Connection Ownership"): the **primary dials out** to standbys; standby-side `ha_find_and_connect_to_primary()` is dead code (declared `ha.h:741`, never called in production). The functional goal (persistent channel, no per-RPC connections) is achieved |
| 6.2 disconnect → immediate election | Implemented | EPOLLRDHUP on `primary_ctrl_fd` → `ha_on_primary_ctrl_fd_rdhup()` (`ha.c:3183`) |
| 6.3 heartbeat resets timeout | Implemented | `ha_on_heartbeat_received()` on every NOP/ctrlhello (`cluster.c:249, 283`) |
| 6.4 heartbeat timeout + jitter → election | Implemented | `ha_arm_heartbeat_timeout_vtask` liveness poll; manual test 11.2 measured ~11s detection (configured timeout, not sub-second — sub-second applies to the RDHUP path) |
| 6.5 sync missing state before admitting to replication set | Implemented | heartbeat-reply-driven driver: standby `stateReport` → `ha_repl_decide()` (`ha.c:1157-1176`) → stream WAL / snapshot / config; `in_catch_up` gates quorum membership (`repl.c:367`) |
| 6.6 snapshot when standby older than oldest WAL segment | Implemented | `standby_lsn < oldest_wal_lsn → HA_REPL_SEND_SNAPSHOT` (`ha.c:1176`), `ha_send_snapshot_to_standby()` chunked transfer (`config_repl.c:1578`), standby handler `cluster.c:182` |
| 6.7 config push after catch-up on tag mismatch | Implemented | same driver pushes config on tag divergence; `push_config_to_standbys` |
| 6.8 winner closes outbound, accepts inbound | Implemented | role flip handled in `raft_cb_become_leader` / RDHUP path; surviving vote eps promoted to `standby_endpoints`; late joiners get fresh outbound connects from `primary_heartbeat_fn` |
### Requirement 7 — Client-Side Transparent Retry (libxce)
| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 7.1 NOT_READY → backoff retry | Implemented | `xce_request_with_retry()` `request.c:1664`; tests `test_libxce_retry.c` |
| 7.2 STANDBY + redirect → follow redirect | **Partial** | Client logic implemented and unit-tested via hook, but **no controller reply ever carries `redirect_host/port`** (see 1.4) — end-to-end the redirect-follow path can never trigger; falls back to backoff |
| 7.3 TCP refused/timeout → reconnect retry | Implemented | `request.c` retry classification |
| 7.4 non-retriable surfaced immediately | Implemented | AUTH/PERM/INVALID/NOENT surfaced without retry |
| 7.5 configurable params + defaults | Implemented | `xce_retry_config_t`, `XCE_RETRY_DEFAULTS` (`request.h:140-150`) |
| 7.6 thundering-herd jitter | Implemented | jittered exponential backoff; 50-client concurrency test |
| (adoption) | **Missing** | Zero production call sites: worker, scheduler, and apiserver backend never call `xce_request_with_retry` — the wrapper is shipped but unused |
### Requirement 8 — Majority-Commit Config Replication
| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 8.1 majority commit for config push | Implemented | `push_config_to_standbys` → `repl_wait_majority_ack(..., OPCode_upgradeConfigAck)` (`config_repl.c:905`) → `committed_config_tag` advanced (`config_repl.c:908`) |
| 8.2 standby durable write + ACK | Implemented | `handle_repl_config_push` + `OP_upgradeConfigAck`; `acked_config_lsn` tracked separately from WAL acks (`repl.c:375-380`) |
| 8.3 timeout → error | Implemented | `-ETIMEDOUT` from `repl_wait_majority_ack`, WARN logged |
| 8.4 promoted standby serves only committed config tag | **Partial** | `last_known_committed_config_tag` captured from CtrlHelloReply (`cluster.c:261`) and `committedConfigTag` piggybacked on heartbeat (capnp); `committed_config_tag` consulted at `ha.c:2514-2521` and safe-tag computed on promotion (`ha.c:485`). Enforcement at *serve time* (refusing to serve a fresher-but-uncommitted config) not explicitly verified |
| 8.5 single-node local write = majority | Implemented | safe-tag path on become-leader; `committed_config_tag` advanced on push without standbys |
### Requirement 9 — Election State Durability
| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 9.1 persist term+vote before reply | Implemented | raft `on_persist` fires before any grant/RPC (`raft.c:77, 176`); `ha_persist_term()` atomic write+rename with magic+CRC32 (`ha.c:1647-1681`) |
| 9.2 restore on startup | Implemented | `ha_restore_term()` (`ha.c:1683+`); manual test 11.4 confirmed `restored term=2` on restart |
| 9.3 missing file → term 0, no vote | Implemented | missing/short-read/bad-magic/CRC-mismatch all degrade safely to term 0 with WARN (`ha.c:1715-1737`) |
### Requirement 10 — Observability
| AC | Status | Assets / Evidence |
|----|--------|-------------------|
| 10.1 election outcome: term + votes granted + quorum size | **Partial** | FSM transition INFO logs carry term (`ha.c:507, 659, 870, 881, 2080, 2158`); majority-ack log carries acks+quorum (`ha.c:2929`); but no single election-complete record logs *votes granted* and *quorum size* together |
| 10.2 bootstrap probe finding | **Partial** | voter-only decision logged (`ha.c:3134`); per-peer probe logs exist in `xce_probe_config_tag` but that path is unused in production |
| 10.3 WAL quorum INFO / timeout WARN | Implemented | `repl.c:408-410` |
| 10.4 config quorum INFO / timeout WARN | Implemented | `config_repl.c` push path logs both |
| 10.5 FSM transition INFO | Implemented | all transitions logged, including RDHUP (`ha.c:3199`) |
### Summary Tally (by requirement)
| Req | Verdict |
|-----|---------|
| R1 | Implemented except redirect payload (1.4 Partial) |
| R2 | Implemented; 2.8/2.10/2.11/2.12 are sanctioned deviations needing spec reconciliation |
| R3 | Pivoted to primary-push; 3.1/3.4 Missing-as-specified, 3.3 Unknown (cold-start convergence untested) |
| R4 | Implemented (4/4) |
| R5 | Core implemented; 5.1 coverage gap, 5.5 CP→degraded-mode deviation, 5.7 restart-path gap, 5.8 needs verification |
| R6 | Implemented; 6.1 direction inverted by design (Constraint) |
| R7 | Library implemented; redirect payload + production adoption Missing |
| R8 | Implemented; 8.4 serve-time enforcement Partial |
| R9 | Implemented (3/3) |
| R10 | Implemented except consolidated election-outcome record (10.1) |
**Net: ~40 of 50 acceptance criteria fully implemented; 7 partial; 3 missing-as-specified (all three superseded or deliberately pivoted).** No requirement is wholly unimplemented.
## 2. Remaining Gaps — Concrete List
1. **G1 — Redirect payload never populated** (R1.4, R7.2): capnp fields exist; standby replies send `nodata`. Client redirect-follow logic is dead end-to-end.
2. **G2 — WAL majority-commit covers only job submission** (R5.1): state transitions and queue changes are async-only; a failover can lose acknowledged state changes.
3. **G3 — Commit timeout is degraded-mode, not an error** (R5.5): spec promises CP (timeout error to caller); code chooses availability. Needs an explicit, documented decision.
4. **G4 — Single-node `committed_lsn` runtime advancement unverified** (R5.8): `-ENODEV` short-circuit skips the advance; interacts with the WAL trim guard.
5. **G5 — Standby restart path skips divergent-tail truncation** (R5.7): truncation happens only in the RDHUP/watchdog path.
6. **G6 — Config probe path unwired** (R3.1/3.3/3.4): `xce_probe_config_tag` is production-dead; all-zero cold-start converges only via watchdog timeout and lacks a test.
7. **G7 — libxce retry has zero production adopters** (R7 objective): worker/scheduler/apiserver still have whatever ad-hoc behavior they had.
8. **G8 — Observability**: no consolidated election-outcome log with vote tally + quorum size (R10.1).
9. **G9 — Spec/code reconciliation debt**: 2.8 (standby instead of fatal exit), 2.10/2.11 (watchdog retry loop instead of retry-once), 2.12 (priority-based stagger), 3.x (push model), 5.5 (degraded mode), 6.1 (inverted channel direction) — all stable code behavior contradicting EARS text.
10. **G10 — Cleanup**: `ha_find_and_connect_to_primary` dead code; bounded per-RDHUP `elect_ctx` leak (`ha.c:3252-3258`, documented UAF-avoidance).
## 3. Implementation Approach Options
### Option A — Extend existing components in place (code-first)
Close G1–G8 directly in `ha.c` / `persist.c` / `repl.c` / `join.c` / worker+scheduler clients; patch requirements.md text afterward.
- Files: `join.c`+`reply.c` (G1), `persist.c` hook routing (G2/G4), `cluster.c` startup replay (G5), `myapp.c`+`config_repl.c` (G6), worker/scheduler request sites (G7), `ha.c` logging (G8).
- ✅ No new modules; follows established patterns (every gap has an existing seam and a testsuite file).
- ✅ Fastest path to closing the spec.
- ❌ G2/G3 are *policy* changes with latency and availability consequences — landing them silently inside this aging spec hides a significant decision.
- ❌ `ha.c` is already 3,818 lines; further growth degrades maintainability.
### Option B — Spec-first reconciliation, then targeted follow-up spec
Amend requirements.md/design.md to match the as-built architecture (push bootstrap, inverted channel, watchdog retry, degraded mode), close only the unambiguous small defects (G1, G4, G8, G10) here, and move G2/G3/G5/G7 into the active `ha-repl-hardening` spec (or a new `ha-cp-commit` spec) with their own requirements review.
- ✅ Honest record: the EARS text stops contradicting shipped behavior; the CP-vs-degraded decision gets explicit operator-facing review.
- ✅ Keeps this spec closeable; G2/G3 get proper TDD treatment in a spec that already owns repl hardening.
- ❌ More process; durability gap (G2) stays open until the follow-up lands.
- ❌ Risk that "deferred" becomes "dropped" without tracking.
### Option C — Hybrid (recommended)
Phase 1 (this spec, S-size code): G1 redirect payload, G4 single-node commit advance verification/fix, G8 election-outcome log, G10 cleanup, plus requirements.md deviation annotations (G9) and a cold-start integration test for G6's watchdog fallback.
Phase 2 (`ha-repl-hardening` spec): G2 (extend sync-append coverage to state transitions/queue changes — likely with group-commit batching to bound latency), G3 (explicit CP-vs-degraded decision + config knob), G5 (restart-path truncation), G7 (retry adoption across daemons).
- ✅ Ships every low-risk fix now; routes every policy-bearing change through review.
- ✅ Matches the existing spec topology (repl-hardening already owns quorum/commit semantics).
- ❌ Requires coordinating two specs; G6's probe-vs-watchdog question must be answered during reconciliation (recommend: delete the probe path, document the watchdog fallback, keep the test).
## 4. Effort & Risk (remaining work)
| Gap | Effort | Risk | Justification |
|-----|--------|------|---------------|
| G1 redirect payload | S | Low | Fields exist in schema/typemap; populate in 2 reply sites + 1 e2e test |
| G2 sync-append coverage | M | Medium | Mechanical routing change but puts majority-RTT on hot job-state path; needs batching design + latency measurement |
| G3 CP-vs-degraded decision | S (code) / M (decision) | High | One-line behavior flip, but it trades availability for consistency cluster-wide; needs operator sign-off and failure-mode tests |
| G4 single-node committed_lsn | S | Medium | Small fix, but wrong answer corrupts WAL trim guard invariant (`committed_lsn <= applied_lsn`) |
| G5 restart-path truncation | M | Medium | Touches startup replay ordering; must honor TLA+ safe-commit-point rule (never truncate via stale ctrlhello value) |
| G6 cold-start test + probe removal | S | Low | Behavior already converges via watchdog; work is test + dead-code deletion + doc |
| G7 retry adoption | M | Medium | Multi-daemon change; behavioral shift in error surfacing for worker/scheduler/apiserver |
| G8 election-outcome log | S | Low | Data already in `ha_elect_ctx_t`/raft state at decision points |
| G9 spec reconciliation | S | Low | Documentation only; six identified deltas |
| G10 cleanup | S | Low | Dead-code deletion + bounded-leak fix behind existing tests |
Aggregate remaining: **~M total for Phase 1 (all S items), ~L for Phase 2** (G2+G3+G5+G7 under ha-repl-hardening).
## 5. Recommendations
1. **Adopt Option C.** Close this spec with Phase 1; transfer G2/G3/G5/G7 into `ha-repl-hardening` as explicit requirements so they get their own approval gate.
2. **Reconcile requirements.md now** (G9): rewrite AC 2.8, 2.10–2.12, 3.1–3.5, 5.5, 6.1 to describe the as-built push/watchdog/outbound-primary architecture. The controller CLAUDE.md and TLA+ mapping are already correct — the spec is the lagging artifact.
3. **Decide G3 deliberately.** The spec's headline promise is "zero data loss on failover" (CP); the code ships degraded-mode availability. Either add a `commit_strict` config knob defaulting per the decision, or amend the spec's CP claim. Do not leave the contradiction implicit.
4. **Fix G1 immediately** — it is the cheapest fix with end-to-end user impact (failover redirect for clients), and it unblocks the already-tested libxce retry path.
5. **Delete dead code** (`ha_find_and_connect_to_primary`, production `xce_probe_config_tag` path if reconciliation confirms the push model) rather than leaving two competing bootstrap narratives in the tree.
### Research Needed
- **RN1**: Does single-node `committed_lsn` ever advance at runtime (G4)? Trace `persist_wal_sync_append` + WAL trim guard in a single-node soak; check whether `ha-repl-hardening` task 6.5 already addressed this.
- **RN2**: Latency budget for extending majority-commit to job state transitions (G2) — measure per-entry RTT on 3-node LAN; evaluate group commit / pipelined acks before committing to a design.
- **RN3**: Restart-path divergence (G5) — construct the failure: standby with uncommitted WAL tail restarts, rejoins, receives catch-up; verify whether `apply_wal_entry` overwrite semantics mask or corrupt the divergent tail.
- **RN4**: All-zero cold-start convergence time and liveness (G6) — 3-node cluster, no configs anywhere; confirm watchdog-driven election converges and measure time-to-primary.
- **RN5**: Serve-time enforcement for 8.4 — can a promoted standby serve a config tag above `committed_config_tag` in any window between promotion and first quorum push?

140
.kiro/specs/ha-heartbeat-liveness/research.md
View file

@ -167,3 +167,143 @@ liveness path, not a new subsystem.
- **ha-heartbeat-liveness** is **~97% complete**. All core implementation is done. The `hb_timeout_thread` has been replaced with delaymgr-based `liveness_poll_fn`.
- **Remaining gaps**: (1) Req 1.3 partial — `OPCode_replEntry` and `OPCode_walCatchupEntry` dispatch sites in ha.c do not call `ha_on_heartbeat_received`. Low risk: active WAL replication keeps `last_push_time` fresh on primary, suppressing NOPs, and ctrlhello resets the standby's deadline. Adding 2 `ha_on_heartbeat_received` calls in the dispatch sites would achieve full literal compliance. (2) Task 4.5 — SIGSTOP/SIGCONT missed-heartbeat tolerance is an optional manual integration test (script exists at `src/tools/test-heartbeat-tolerance.sh`).
- **Cross-spec relationship**: `ha-repl-hardening` heartbeat-driven convergence tracking relies on this spec's liveness infrastructure. The `primary_heartbeat_fn` idle-triggered NOP suppression is correct and does not interfere with ha-repl-hardening's per-standby state reporting via heartbeat replies.
---
# Gap Analysis — Validation Pass (2026-06-09)
> Scope: re-validation of ha-heartbeat-liveness on the persistent_ha working tree
> (uncommitted changes to ha.c/ha.h/myapp.c/raft.c analyzed as on disk), after
> commit e1517bb "heartbeat re-arm broken by delaymgr ms-task ordering".
## 1. Current State Investigation
All tasks in tasks.md are checked done (1.1–4.5). On-disk verification:
| Asset | Location | Verified state |
|-------|----------|----------------|
| `liveness_poll_ctx_t` + `_Atomic cancelled` | `src/controller/ha.c:3308` | Present; `s_liveness_ctx` is `_Atomic` ptr (ha.c:3320) |
| `liveness_poll_fn` | `ha.c:3333` | cancelled check → role guard → elapsed vs `heartbeat_timeout_ms` → `ha_on_primary_ctrl_fd_rdhup` or re-arm at `interval/2` (min 100ms) |
| `ha_arm_heartbeat_timeout_vtask` | `ha.c:3408` | delaymgr-based; disarms stale ctx, seeds `last_activity = now`, INFO log |
| `ha_on_heartbeat_received` | `ha.c:3453` | writes `CLOCK_MONOTONIC` secs to `ha->last_activity`; DEBUG3 log |
| `ha_disarm_heartbeat_timeout_vtask` | `ha.c:3462` | sets cancelled, clears static ptr; poll fn frees ctx |
| RDHUP cancels poll (Req 5.4) | `ha.c:3189` | `ha_disarm_heartbeat_timeout_vtask` is the first action in `ha_on_primary_ctrl_fd_rdhup` |
| `primary_heartbeat_fn` idle-NOP | `ha.c:3692–3740` | per-standby `last_push_time` elapsed check before `xce_nop_req` |
| `ha_start_primary_heartbeat` | `ha.c:3784` | post-e1517bb: `runcount=-1` infinite delaymgr task + static `_Atomic` once-guard; no fragile re-arm path |
| delaymgr ordering fix | `src/lib/delay.c` (e1517bb) | `task_cmp` now orders ms-tasks by `nextrun_ts` (ns precision) — re-arm starvation root cause fixed |
| `validate_heartbeat_config` | `src/controller/startup_validation.c:143` | `timeout >= 2×interval` else `-EINVAL` + message; wired in `cluster.c:885` before `ha_init()` (design said main.c; cluster.c init path is equivalent) |
| Opcode liveness resets | `cluster.c:250` (ctrlhello), `cluster.c:285` (NOP), `config_repl.c:1373` (handle_ctrlhello) | only these three production call sites of `ha_on_heartbeat_received` |
| Arm call sites (Req 5.6) | `ha.c:664` (raft_cb_become_follower), `myapp.c:302` (skip-election standby path), `config_repl.c:1374` (handle_ctrlhello) | all three set/verify CTRL_STANDBY before arming; poll fn re-checks role |
| `hb_timeout_thread` / `hb_vtask_t` / `s_hb_vtask` | — | removed; only historical comments remain |
| Unit tests | `testsuite/test_ha_liveness.c`, `test_ha_startup.c:712ff` | present, registered in `meson.build:120–121` |
| Integration | `testsuite/test_ha_rdhup_integration.c`, `src/tools/test-heartbeat-tolerance.sh` | present (4.5 script exists) |
Uncommitted working-tree changes to ha.c are election-init refactors (`ha_init_raft`
extraction, term-scoped vote dedup, `primary_ctrl_fd >= 0` vote-gate guard) from the
ha-repl-hardening stream; they do not alter the liveness mechanics of this spec.
## 2. Requirement-to-Asset Map
| Req | Summary | Asset(s) | Status |
|-----|---------|----------|--------|
| 1.1 | Record last-activity timestamp | `ha_state_t.last_activity` (ha.h; init ha.c:814; writer ha.c:3453) | Implemented |
| 1.2 | Periodic elapsed check | `liveness_poll_fn` ha.c:3333; poll = `interval/2`, floor 100ms | Implemented |
| 1.3 | ANY opcode resets deadline | NOP + ctrlhello only; `OPCode_walCatchupEntry` (cluster.c:407), `OPCode_replConfigPush`, `OPCode_snapshotTransfer` handlers do NOT call `ha_on_heartbeat_received` | **Partial** — compensated at runtime because NOPs are not actually suppressed during WAL traffic (see 3.3) |
| 1.4 | Failover on timeout | ha.c:3367 → `ha_on_primary_ctrl_fd_rdhup` | Implemented |
| 1.5 | Stop poll on role change | disarm in RDHUP (ha.c:3189) + role guard in poll fn (ha.c:3356) | Implemented (minor: role-guard early-return at ha.c:3357 returns without freeing/cancelling ctx — a one-time ~32-byte leak on the rare promote-without-disarm path) |
| 1.6 | No blocking thread | `hb_timeout_thread` removed | Implemented |
| 2.1 | Dead only after `heartbeat_timeout_ms` | poll threshold ha.c:3365–3367 | Implemented — **Constraint**: whole-second arithmetic (`timeout_s = ms/1000`, `elapsed` in truncated secs) can fire up to ~1s off the ms-precise spec; sub-second intervals unsupported |
| 2.2 | 3000/9000 tolerates 2 misses | config convention + `test-heartbeat-tolerance.sh` | Implemented |
| 2.3 | 3000/6000 tolerates 1 miss | same | Implemented |
| 2.4 | Startup validation | `validate_heartbeat_config` + cluster.c:885 wiring + tests (test_ha_startup.c:727–759) | Implemented |
| 3.1 | Track per-standby outbound idle | `standby_ep_ctx_t.last_push_time` ha.c:3490 | **Partial** — written only at `on_standby_connected` (ha.c:3561) and after NOP (ha.c:3735) |
| 3.2 | NOP after one idle interval | ha.c:3725–3736 | Implemented |
| 3.3 | ANY send resets idle timer | WAL batch (`send_wal_batch`), config push, snapshot send paths never write `last_push_time` (grep confirms only 2 write sites) | **Missing** — design.md:275 deferred the repl.c decision and it never landed |
| 3.4 | No redundant NOP within interval | elapsed check exists, but because 3.3 is missing, NOPs continue at full rate during active WAL/config/snapshot traffic | **Partial** — benign deviation (extra NOPs, no correctness impact; in fact it is what keeps 1.3's gap safe) |
| 3.5 | Stop idle task on demotion | role guard ha.c:3699; `runcount=-1` + `_Atomic` once-guard (ha.c:3796) | Implemented |
| 4.1 | ctrlhello counts as activity | cluster.c:250 + config_repl.c:1373 | Implemented |
| 4.2 | Committing window safe | arm + `last_activity=now` at ctrlhello; threshold = full timeout | Implemented |
| 4.3 | No premature declare within window | poll threshold; subject to ±1s granularity constraint above | Implemented |
| 5.1 | Raft unchanged | raft.c untouched **by this spec** (working-tree raft.c edits belong to ha-repl-hardening) | Implemented |
| 5.2 | RDHUP immediate fast path | handler unchanged; liveness only covers silent-dead | Implemented |
| 5.3 | Both paths → same failover fn | poll fn calls `ha_on_primary_ctrl_fd_rdhup` | Implemented |
| 5.4 | RDHUP cancels pending poll | ha.c:3189 disarm-first | Implemented |
| 5.5 | Single-node non-interference | empty `standby_endpoints` loop is no-op; reconnect loop skips self | Implemented |
| 5.6 | No arm/exec in INIT/ELECTING | all 3 arm sites are standby-only; poll fn role guard | Implemented |
| 5.7 | INFO log arm/reset/fire | arm = INFO (ha.c:3442); fire = WARN (ha.c:3369); reset = DEBUG3 (ha.c:3459) | **Partial** — reset is DEBUG3 by deliberate documented design deviation (log-spam avoidance); fire is WARN not INFO (arguably more correct) |
**Score: 20 of 25 acceptance criteria fully Implemented; 4 Partial (1.3, 3.1, 3.4, 5.7); 1 Missing (3.3).**
The post-e1517bb runtime behavior satisfies the feature's operational intent: heartbeats
re-arm correctly (delaymgr `task_cmp` ordering fixed + infinite-repeat task), no false
failovers in the committing window, tolerance via timeout ratio works.
### New cross-cutting defect found (out-of-boundary but caused by this refactor)
**`ha_heartbeat_is_fresh` (ha.c:1762) reads a dead field.** It computes elapsed from
`ha->last_heartbeat` (struct timespec, ha.h:154), but no production code ever writes
`last_heartbeat` — only tests do (`test_ha_vote_gate.c`). The liveness refactor moved all
activity tracking to `ha->last_activity` (design.md data model even lists `last_heartbeat`
as "Repurposed / replaced") yet the vote-deny gate was never repointed. Consequence: once
process uptime exceeds `heartbeat_timeout_ms` (~10s), `ha_heartbeat_is_fresh` always
returns stale, so the "STANDBY with live primary denies votes" gates at **ha.c:214**
(raft vote callback) and **ha.c:1841** (`handle_request_vote`) are inert. A disturber
candidate can collect votes from standbys whose primary is healthy and usurp leadership.
The uncommitted `primary_ctrl_fd >= 0 &&` guard at ha.c:1841 narrows the deny condition
further (it does not restore it), and the ha.c:214 site has no fd guard at all.
**Fix is a one-liner**: compute freshness from `ha->last_activity` (and either delete
`last_heartbeat` or update the vote-gate tests). Tag: **Missing** (vote-gate liveness
input), Severity: Medium (availability — unnecessary election churn; not data loss, since
term/log freshness gates still apply).
### Secondary observation — type-punned idle tracking on promoted vote endpoints
`primary_heartbeat_fn` casts `ep->data` to `standby_ep_ctx_t*` for every entry in
`standby_endpoints`, but promoted vote endpoints carry `vote_ep_ctx_t` (ha.c:45) whose
5th field is `uint64_t my_term` — which aliases `last_push_time` (both at offset 32).
Reading it yields a tiny term number → elapsed always huge → NOP always sent on the first
tick, then `now` is written into `my_term`. Functionally self-correcting, but it silently
clobbers `my_term` and depends on struct-layout coincidence. Tag: **Constraint** (fragile;
unify the ctx structs or skip idle tracking when `sctx->ctx != NULL`).
## 3. Implementation Approach Options for Remaining Gaps
### Option A — Targeted point fixes in existing files (recommended)
1. `ha_heartbeat_is_fresh`: read `ha->last_activity`; adjust `test_ha_vote_gate.c`; drop or deprecate `ha_state_t.last_heartbeat`. (~10 lines + test updates)
2. Req 1.3 literal compliance: add `ha_on_heartbeat_received(&c->ha)` (role-guarded) to the `OPCode_walCatchupEntry`, `OPCode_replConfigPush`, `OPCode_snapshotTransfer` dispatch arms in cluster.c. (~6 lines)
3. Req 3.3 (optional): write `last_push_time` in the primary's WAL/config/snapshot send helpers, or explicitly document extra-NOPs-during-replication as the accepted behavior.
- ✅ Smallest diff, stays inside spec boundary files, leverages existing tests
- ❌ Leaves the `vote_ep_ctx_t`/`standby_ep_ctx_t` aliasing in place
### Option B — Centralized channel-activity layer
Introduce a small `ha_liveness` helper that hooks endpoint send/recv once (lib layer) and owns both `last_activity` and per-ep `last_push_time`; unify the two ep ctx structs.
- ✅ Eliminates per-opcode wiring forever; fixes the aliasing structurally
- ❌ Touches `src/lib` endpoint layer (outside spec boundary), M effort, new revalidation triggers for three other HA specs
### Option C — Hybrid (A now, struct unification later)
Apply Option A immediately; file the ctx-struct unification and Req 3.3 send-path tracking as a follow-up item under ha-repl-hardening where repl.c is already in boundary.
- ✅ Unblocks completion with low risk; defers structural work to a spec that owns repl.c
- ❌ Two-step bookkeeping; deviation on 3.3/3.4 persists in the interim
## 4. Effort & Risk
| Item | Effort | Risk | Justification |
|------|--------|------|---------------|
| Fix `ha_heartbeat_is_fresh` → `last_activity` | S | Low (fixes a Medium-severity defect) | one-line logic change + 2 test helpers; existing vote-gate tests pin behavior |
| Req 1.3 opcode wiring (3 handlers) | S | Low | identical pattern to existing NOP/hello call sites |
| Req 3.3 send-path `last_push_time` | S | Low–Medium | needs ep→sctx access in config_repl.c send helpers; aliasing caveat applies |
| ctx struct unification (aliasing) | M | Medium | touches election ep promotion path; needs 3-node regression |
| Overall remaining work | **S** (≤1 day for Option A) | **Low** | all patterns established; tests and deploy harness exist |
## 5. Recommendations & Research Needed
**Recommended approach: Option C** — land Option A's three point fixes now (the
`ha_heartbeat_is_fresh` repoint should go in before the current working tree is
committed, since the uncommitted vote-gate edit at ha.c:1841 builds on the broken
freshness check), and move struct unification + send-path idle tracking into
ha-repl-hardening's backlog.
**Research Needed:**
- R1: Is `vote_ep_ctx_t.my_term` read anywhere after an endpoint is promoted into `standby_endpoints`? (If yes, the `last_push_time` clobber is an active bug, not just fragility.)
- R2: Are sub-second `heartbeat_interval_ms` configurations a supported operating point? If yes, `liveness_poll_fn`/`primary_heartbeat_fn` must move from whole-second to ms arithmetic (`timespec` diff).
- R3: Confirm with ops/runbook owners that DEBUG3-on-reset (vs Req 5.7's literal INFO) is the accepted trade-off; if so, amend Req 5.7 wording.
- R4: Verify task 4.5 (`test-heartbeat-tolerance.sh`) has been executed against the post-e1517bb binary; the delaymgr ordering fix changed scheduling behavior after the last recorded run.

97
.kiro/specs/ha-peer-connection-reuse/research.md
View file

@ -235,3 +235,100 @@ All seams appear to be in place. The tests should pass, but a build run (`/xce-b
- **Integration tests (7.2, 7.3)**: Both marked `[x]`. The two-node integration test and WAL ACK test are done.
- **Remaining**: Build verification (`/xce-build controller`) to confirm all unit tests in `test_ha_connection_reuse.c` pass under ASAN.
- **Cross-spec relationships**: `ha-repl-hardening` commit `0762577` (task 5.1: "reduce handle_ctrlhello to connection setup only") directly depends on this spec's `peer_ctrl_msg_handler` + promoted connection model. The two specs are consistent — ha-repl-hardening moves all replication decisions to the heartbeat decision loop, which works over the promoted connections established by this spec.
---
# Gap Analysis — Validation Pass (2026-06-09)
> Scope: requirements.md (Reqs 1–5) vs working tree on `persistent_ha` (including uncommitted changes to `ha.c`, `ha.h`, `myapp.c`, `raft.c`). All tasks in tasks.md are checked `[x]`. This pass re-verifies each acceptance criterion against on-disk code and a live run of the controller unit-test binary.
## 1. Current State Summary
- All seven task groups are implemented in `src/controller/ha.c`, `src/controller/ha.h`, `src/controller/config_repl.c`, with tests in `src/controller/testsuite/test_ha_connection_reuse.c` (registered in `src/controller/meson.build:119`).
- The dispatcher (`peer_ctrl_msg_handler`, ha.c:2570) is set once in `on_vote_connected` (ha.c:2321, 3586) and routes by `ha->role`: ELECTING → `vote_reply_decode_and_dispatch` (ha.c:2370), COMMITTING/PRIMARY → `repl_ack_decode_and_dispatch` (ha.c:2456), other → WARN + drop (ha.c:2589).
- Winner promotion: `raft_cb_become_leader` (ha.c:513–529) appends every non-NULL `votes[i].ep` to `ha->standby_endpoints` under `ha->lock`, then pushes config (ha.c:544) and ctrlhello (ha.c:559) over those endpoints.
- Loser path: `raft_cb_become_follower` (ha.c:640–664) calls `endpoint_fini` on each vote ep, sets `CTRL_STANDBY`, then arms `ha_arm_heartbeat_timeout_vtask` — no outbound activity afterward.
- Standby side: `handle_ctrlhello` (config_repl.c:1357–1374) sets `primary_ctrl_fd = ep->fd` (idempotent `== -1` guard) and resets the liveness watchdog via `ha_on_heartbeat_received`.
- The tier-1/2/3 discovery path is fully removed: `ha_find_and_connect_to_primary` and `fetch_config_from_primary` no longer exist as functions (only stale comment references remain — ha.h:741, ha.c:3136–3137, cluster.c:1006, main.c:168). myapp.c:289 documents the new passive-standby contract.
- Implementation went beyond the spec in one compliant way: late joiners are reconnected by the primary heartbeat task (`primary_heartbeat_fn` / `on_standby_connected`, ha.c:85–86) via fresh outbound connections — this respects Req 4.2's "in that election round" restriction.
- Live test run (controller_unit_test built 2026-06-09 13:07, newer than all sources): 769 PASS, 0 FAIL; all 10 connection-reuse tests pass. However ASAN reports **37,360 bytes leaked in 397 allocations** at process exit (see Req 5.5 below).
## 2. Requirement-to-Asset Map
| Req | Summary | Code Assets | Status |
|-----|---------|-------------|--------|
| 1.1 | One outbound TLS connection per peer for vote RPC | `raft_cb_send_vote_request` ha.c:~147–190; SSL via `ha->client_ssl_ctx` | **Implemented** (Constraint: TLS only when `client_ssl_ctx` is configured; plaintext fallback otherwise) |
| 1.2 | Reuse surviving vote connection as persistent peer channel | `peer_ctrl_msg_handler` ha.c:2570; set once at ha.c:2321/3586, never reassigned | **Implemented** — test `test_dispatcher_*` x3 PASS |
| 1.3 | Winner promotes vote connections to standby channels | `raft_cb_become_leader` promotion loop ha.c:513–529 | **Implemented** — `test_become_leader_promotes_vote_endpoints`, `..._skips_null_endpoints` PASS |
| 1.4 | Loser closes all outbound vote connections | `raft_cb_become_follower` close loop ha.c:640–655 | **Implemented** — `test_become_follower_closes_vote_endpoints` PASS |
| 2.1 | Standby initiates no outbound TCP to peers | `fetch_config_from_primary` deleted; passive wait myapp.c:288–300 | **Implemented** — `test_follower_passive_no_outbound_connect` PASS |
| 2.2 | Standby processes ctrlhello/config/WAL on inbound connection | cluster.c:246 dispatch → `handle_ctrlhello`; existing config/WAL handlers unchanged | **Implemented** |
| 2.3 | No inbound comm within timeout → new election | `ha_arm_heartbeat_timeout_vtask` ha.c:3408 armed at ha.c:664 and myapp.c skip-election path; fires `ha_on_primary_ctrl_fd_rdhup` → CTRL_ELECTING | **Implemented** (Note: comment at ha.c:662–663 says "election_timeout_ms"; actual watchdog uses `heartbeat_timeout_ms` — stale comment) |
| 2.4 | No tier-1/2/3 discovery on standby | `ha_find_and_connect_to_primary` removed entirely (exceeds design, which said it would "remain for future use") | **Implemented** |
| 3.1 | First ctrlhello designates primary control channel + arms watchdog | config_repl.c:1357–1374, idempotency guard `primary_ctrl_fd == -1` | **Implemented** — `test_handle_ctrlhello_sets_primary_ctrl_fd` PASS |
| 3.2 | RDHUP detected within one epoll cycle → CTRL_ELECTING | `ha_on_primary_ctrl_fd_rdhup` ha.c:3183–3199; iomgr err-handler path unchanged | **Implemented** |
| 3.3 | Watchdog same timeout semantics as `primary_ctrl_fd` | Same `ha_arm_heartbeat_timeout_vtask` vtask, `heartbeat_timeout_ms` check ha.c:3302+ | **Implemented** |
| 4.1 | Explicit close of vote endpoints on loss | `endpoint_fini` + NULL-out, ha.c:648–652 | **Implemented** |
| 4.2 | Failed vote connection → denied vote, no reconnect that round | `vote_ep_err_handler` ha.c:2942: ELECTING → `handle_request_vote_reply(grant=0)`; COMMITTING/PRIMARY → remove ep from `standby_endpoints` under lock | **Implemented** — `test_err_handler_removes_promoted_ep_from_standby` PASS |
| 4.3 | New election → fresh outbound connections | `ha_start_election_async` allocates fresh `ha_elect_ctx_t` per round; eps created in `raft_cb_send_vote_request` | **Implemented** |
| 5.1 | Config push via promoted connections | ha.c:531–553 `push_config_to_standbys` over populated `standby_endpoints` | **Implemented** |
| 5.2 | WAL replication via promoted connections | ctrlhello → catch-up over promoted eps; `repl_ack_decode_and_dispatch` → `ha_repl_ack_received` | **Implemented** — `test_promoted_ep_repl_ack_promotes_to_primary` PASS |
| 5.3 | Single-node self-elect, no peer connections | raft library shortcircuit (unchanged); `ectx`/NULL guards in both callbacks | **Implemented** |
| 5.4 | Committing timeout → degraded PRIMARY | `committing_timeout_fn` armed ha.c:567–591; cancel-on-stepdown ha.c:611–621 | **Implemented** |
| 5.5 | No new ASAN errors vs baseline | Live run: 769 PASS / 0 FAIL, but ASAN leak report at exit: 37,360 B / 397 allocs. Traces include this spec's test file (fixture callocs at test_ha_connection_reuse.c:334–335, 470–471 — test-only) AND one production allocation: `ha_arm_heartbeat_timeout_vtask` ctx (ha.c:3424) leaked when invoked via `handle_ctrlhello` (config_repl.c:1374) in the test harness (no live delaymgr to consume/free the ctx) | **Partial** — leaks exist; whether they are "new vs baseline" is unverified (Unknown) |
**Score: 18 of 19 acceptance criteria Implemented; 1 Partial (5.5).**
### Documentation/metadata drift (not functional gaps)
- ha.c:3136–3137 comment still says "myapp_run schedules fetch_config_from_primary" — the function no longer exists.
- ha.h:741 comment references removed `ha_find_and_connect_to_primary`.
- ha.c:662 watchdog comment says `election_timeout_ms`; code uses `heartbeat_timeout_ms`.
- design.md places `handle_ctrlhello` in `cluster.c`; the actual handler lives in `config_repl.c:1357` (tasks.md is correct).
- design.md says `ha_find_and_connect_to_primary` "remains for future use"; it was removed entirely (the stronger guarantee — acceptable, but design should be noted as superseded).
- spec.json `phase` is still `"tasks-generated"` although all tasks are `[x]` and tests pass.
- Working tree carries uncommitted changes (`ha_init_raft` extraction, vote-gate now requires `primary_ctrl_fd >= 0` for the fresh-heartbeat deny, raft.c changes). These interact with this spec's assets and pass tests, but are not committed — provenance vs adjacent specs (ha-repl-hardening / ha-config-bootstrap) is unclear.
## 3. Implementation Approach Options (for the remaining gap)
### Option A: Extend Existing (close Req 5.5 + comment hygiene in place) — RECOMMENDED
- Free test fixtures in `test_ha_connection_reuse.c` (callocs at lines 334–335, 470–471, and ctrlhello test path) at test teardown.
- For the `ha_arm_heartbeat_timeout_vtask` ctx leak: either free the armed ctx in the test teardown via the existing cancel flag, or gate the arm behind the existing `c == NULL` test seam in `handle_ctrlhello`'s call.
- Fix the four stale comments; update spec.json phase.
- ✅ Smallest diff, no interface change, all in files this spec already owns. ❌ Leaves "baseline comparison" implicit — pair with the Research item below.
### Option B: New Components (ASAN baseline harness)
- Add a `tools/asan-baseline.sh` that runs the controller suite on a pinned baseline commit and diffs leak reports, making Req 5.5 mechanically checkable for every spec.
- ✅ Permanent, reusable verification for all HA specs' "no new ASAN errors" criteria. ❌ New tooling outside this spec's boundary; more effort than the gap warrants on its own.
### Option C: Hybrid
- Do Option A now (fix the known leaks regardless of baseline status), and file Option B as a follow-up task under a tooling/steering spec.
- ✅ Closes 5.5 definitively (zero leaks ⇒ trivially "no new errors") while seeding durable verification. ❌ Slightly more coordination.
## 4. Effort & Risk
| Item | Effort | Risk | Justification |
|------|--------|------|---------------|
| Test-fixture leak frees (test file only) | S | Low | Mechanical `free()`/teardown additions in one test file |
| Watchdog-ctx leak under test harness | S | Low | Existing cancel-flag/test-seam patterns apply; production path (delaymgr consumes ctx) unaffected |
| Stale comment + design.md/spec.json corrections | S | Low | Documentation-only |
| Commit hygiene for uncommitted ha.c/ha.h/myapp.c/raft.c changes | S | Medium | Changes span two specs' boundaries; mis-attribution complicates future validation passes |
| Option B baseline-diff tooling (optional) | M | Low | New script + CI wiring, no production code |
**Overall remaining effort: S. Overall risk: Low.**
## 5. Recommendations for Design/Closure
1. **Adopt Option A (or C)**: eliminate the reported leaks rather than litigating the baseline — a leak-free run satisfies Req 5.5 unconditionally.
2. Verify the production-side lifetime of the `ha_arm_heartbeat_timeout_vtask` ctx when `handle_ctrlhello` re-arms repeatedly (cancel-flag of the previous ctx must lead to a free in the delaymgr callback; confirm no slow accumulation on long-lived standbys).
3. Update spec.json to an implementation-complete phase and annotate design.md with the two superseded statements (handle_ctrlhello location; full removal of `ha_find_and_connect_to_primary`).
4. Commit or re-home the uncommitted working-tree changes with spec attribution before the next validation pass.
### Research Needed
- **ASAN baseline diff**: run the controller suite at the pre-feature commit to confirm which of the 397 leaked allocations predate this spec (only needed if Option A is rejected).
- **Watchdog ctx lifetime in production**: trace `heartbeat_timeout_ctx` ownership through cancel/re-arm cycles on a long-lived standby (ha.c:3337–3449).
- **Uncommitted raft.c changes**: determine which spec owns the `ha_init_raft` extraction and the `primary_ctrl_fd >= 0` vote-gate tightening (interacts with Req 3.1's designation semantics), and validate against the TLA+ checklist in `src/controller/CLAUDE.md`.

176
.kiro/specs/ha-repl-hardening/research.md
View file

@ -148,3 +148,179 @@ Add `OPCode_stateReport` / `OPCode_stateAck` to capnp schema:
- **Cross-spec relationships**: This spec extends `ha-config-bootstrap` Req 5 (majority-commit WAL) and Req 8 (majority-commit config) with heartbeat-driven convergence. It depends on `ha-heartbeat-liveness` for the idle-triggered NOP policy on the primary and activity-based liveness detection on the standby. It depends on `ha-peer-connection-reuse` for the promoted connection model — `handle_ctrlhello` was reduced to connection setup only (task 5.1), with all replication decisions moved to the heartbeat decision loop.
- **Single-node mode**: Verified working — `n_peers=1`, immediate convergence, `CTRL_PRIMARY` with no heartbeat wait.
- **Recommendation**: Run `/xce-build controller` to confirm ASAN-clean build, then mark the spec `implemented`.
---
# Gap Analysis — Validation Pass (2026-06-09)
> Validation-phase gap analysis: requirements.md vs code **as on disk** (including uncommitted
> changes to ha.c/ha.h/myapp.c/raft.c). All 27 subtasks in tasks.md are `[x]`; this pass verifies
> what the code actually does against each acceptance criterion.
## Headline Finding
**The control plane is complete; the data plane is not.** Every protocol mechanism in this spec
(heartbeat stateReport, per-standby tracking, decision loop, batch framing, snapshot chunking,
convergence-gated promotion, safe commit point) is implemented and unit/integration tested. But
**payload bytes are never transmitted or applied**:
- `ha_send_snapshot_to_standby` sends every chunk with `chunk.payload = NULL` —
`config_repl.c:1638` (`/* encoding deferred */`, explained at config_repl.c:1613-1618: the
capnp `Data` ↔ `xce_data_t` tagged-union encode path is not wired).
- The standby snapshot handler passes `payload_bytes=NULL, payload_len=0` —
`cluster.c:191-197`. Consequently `snap.part` is never written, the `is_last` size check
(`ha.c:1094-1106`) fails (or `stat` fails on a missing file), and `persist_load_snapshot` is
never reached in production. Snapshot transfer works only in unit tests via
`xce_snapshot_chunk_hook`.
- WAL catch-up applies entries **header-only**: `ha_apply_single_entry` (ha.c:974-984) builds
`payload_len=0` and calls `apply_wal_entry(p, &hdr, NULL)`. `apply_wal_entry`
(persist.c:691) then only advances the LSN chain — payload-bearing entry types
(`WAL_JOB_SUBMIT` → capnp job, jobstate entries → `wal_jobstate_t`) restore **nothing**.
**Safety implication**: a standby's `applied_lsn` advances without the corresponding state, so it
reports "converged" via stateReport, the primary promotes out of COMMITTING, and a later failover
to that standby would serve from a state missing all payload-bearing entries. The Majority
Consistency Invariant (design §Overview) is currently tracked but not actually established.
**Cross-spec constraint**: the live replication path has the same hole — `repl_broadcast`
(repl.c:193-234) discards `payload` and sends nothing ("TCP send wiring deferred to task 7.6",
owned by ha-config-bootstrap). Today no WAL payload bytes move between controllers on any path.
## Requirement-to-Asset Map
| Req / AC | Asset (file:line) | Status |
|---|---|---|
| **1.1** heartbeat reply carries (tag, lsn) | `cluster.c:273-287` OPCode_nop handler; `ha_nop_reply_state_report` ha.c:1140; `StateReport` controller.capnp:788 | Implemented |
| **1.2** primary records + evaluates | `repl_ack_decode_and_dispatch` ha.c:2456-2547 → `ha_process_standby_state` ha.c:1198 | Implemented |
| **1.3** convergence criteria (tag≥ AND lsn≥) | ha.c:1211-1212 | Implemented |
| **1.4** tag=0 before config push | reads `myconf->config_version_tag` (0 pre-push) | Implemented |
| **2.1** init tracking on COMMITTING entry | Arrays zeroed only in `ha_init` (ha.c:816-819 comment claims the invariant); `raft_cb_become_leader` (ha.c:445-529) resets `repl_ack_count` but **not** `standby_converged[]`/`standby_reported_*[]` → stale convergence flags survive a demote→re-elect cycle | **Partial** |
| **2.2** update + re-evaluate on reply | ha.c:1198-1223, 2525-2533 | Implemented |
| **2.3** majority threshold ceil(n/2)−1 | `ha_check_majority_convergence` ha.c:1244-1278 requires **all** standbys (`n_converged >= n_peers−1`), not majority. Three conflicting formulas: requirements.md 2.3 = ceil(n/2)−1 (1 standby for n=3); tasks.md 5.2 = ceil(n/2) (2 for n=3); code = n−1 (2 for n=3, 4 for n=5) | **Constraint / deviation** (stricter than spec — safe but availability hit) |
| **2.4** INFO on promotion | ha.c:1266-1267 | Implemented |
| **2.5** regression → not-converged | ha.c:1210-1213 | Implemented |
| **3.1** decode + apply WAL entry | Handler wired (`cluster.c:161-173` → `ha_apply_wal_catchup_entry` ha.c:986) but applies **header-only, payload dropped** (ha.c:974-984) | **Partial (CRITICAL)** |
| **3.2** advance applied_lsn | `ha_advance_applied_lsn` ha.c:960 (atomic, monotonic) | Implemented |
| **3.3** ERROR + skip on failure | ha.c:1003-1016, 1032-1037 | Implemented |
| **3.4** stays CTRL_STANDBY, no job ops | `ha_dispatch_gate` CTRL_STANDBY case ha.c:1456-1483 | Implemented |
| **3.5** LSN order | sequential batch iteration | Implemented |
| **4.1** create checkpoint + transfer | `ha_send_snapshot_to_standby` config_repl.c:1578 chunks an **existing** `snap-{checkpoint_lsn}.capnp` — never calls `persist_checkpoint()` to create a fresh one; payload=NULL | **Partial** |
| **4.2** load snapshot, set applied_lsn | `ha_handle_snapshot_chunk` ha.c:1058-1131 complete (verify/rename/load/applied_lsn) but unreachable in production (NULL payload, cluster.c:196) | **Partial** |
| **4.3** no job ops until loaded | dispatch gate (standby role unchanged) | Implemented |
| **4.4** retry via reconnect | Restart-on-reconnect documented (ha.c:1054, cluster.c:178-180) but **no production code unlinks `snap.part`** on reconnect — stale partial file breaks the next transfer's size check | **Partial** |
| **4.5** single-node: no snapshot | no standbys → decision loop never fires | Implemented |
| **5.1** multiple entries per message | `entries :List(WalCatchupEntry)` controller.capnp:850; `send_wal_batch` config_repl.c:1409 | Implemented |
| **5.2** configurable max batch size | `WAL_STREAM_BATCH_ENTRIES`=256 / `WAL_STREAM_BATCH_BYTES`=64KB are compile-time #defines, not runtime config | Implemented (minor constraint) |
| **5.3** apply in order, advance after batch | ha.c:992-1027 (`last_good_lsn` post-batch advance) — but payload gap applies | **Partial** |
| **5.4** partial batch sent immediately | config_repl.c:1556-1563 | Implemented |
| **5.5** ordering + durability | Ordering yes; "durably written" weak — entities restored in-memory, no fsync on standby (applied_lsn in-memory-only is a documented design choice), and payload gap applies | **Partial** |
| **6.1** heartbeats continue in COMMITTING | self-re-arming heartbeat task (fix e1517bb) | Implemented |
| **6.2** NOT_READY for non-repl opcodes | `ha_dispatch_gate` CTRL_PRIMARY_COMMITTING allowlist ha.c:1436-1454 | Implemented |
| **6.3** immediate promotion on majority | ha.c:2529-2533 (before repl dispatch) — subject to 2.3 threshold deviation | Implemented (constraint) |
| **6.4** timeout → WARN + degraded mode | `committing_timeout_fn` ha.c:2885-2914; default 60000 (`HA_DEFAULT_COMMITTING_TIMEOUT_MS` ha.h:131) | Implemented |
| **6.5** single-node immediate | n_peers=1 → threshold 0 (ha.c:1261-1264; fixed in e10cf9a) | Implemented |
| **7.1** catch-up start INFO | config_repl.c:1445-1447 | Implemented |
| **7.2** converged INFO | ha.c:1216-1222 | Implemented |
| **7.3** snapshot start/complete INFO | config_repl.c:1595, 1663 | Implemented |
| **7.4** majority INFO | ha.c:1266 | Implemented |
| **7.5** timeout WARN w/ counts | ha.c:2905-2908 | Implemented |
| **Flow 0** safe commit point (tasks 4.1/4.2) | `ha_compute_safe_committed_lsn` ha.c:373-432; truncation + committed baseline in `raft_cb_become_leader` ha.c:456-485 | Implemented |
**Score**: 26 of 33 acceptance criteria fully implemented. Requirements 1, 6, 7 (and Flow 0
safety) Implemented; Requirement 2 Partial (init-reset + threshold semantics); Requirements 3, 4,
5 Partial — all blocked on the same payload data-plane gap.
**Uncommitted working-tree changes** (ha.c/ha.h/myapp.c/raft.c/deploy-ha.sh) are election-protocol
hardening adjacent to this spec: `ha_init_raft` one-time init (prevents raft state reset on
re-election), term-scoped candidate-agnostic vote denial in `raft_on_vote_request` with idempotent
re-grant, startup self-vote/stale-vote handling, and watchdog arming on the skip-election path.
These strengthen Election Safety per raft-tla-audit.md and should be committed, but they do not
close any of this spec's acceptance criteria.
## Implementation Approach Options (remaining gaps)
### Gap 1 — Payload encoding for `WalCatchupEntry.payload` and `SnapshotTransfer.payload` (the blocker)
**Option A — Extend the typemap codegen to map capnp `Data` ↔ raw buffer for these structs.**
The schema already declares `payload :Data` on both structs; only the C-side encode/decode of the
`xce_data_t` tagged union is missing.
- ✅ Fixes both messages at once; follows the established schema/typemap convention; benefits any
future Data-carrying message
- ❌ Touches generated-code path — regression risk across all opcodes; needs codegen expertise
**Option B — Hand-rolled encode/decode for just these two opcodes.**
Bypass the tagged union: serialize `lsn/type/len/bytes` with direct capn calls (or a length-prefixed
raw layout) in `send_wal_batch` / `ha_send_snapshot_to_standby` and their decode counterparts.
- ✅ Localized, no codegen risk, fastest path to a working data plane
- ❌ Two bespoke wire paths diverging from every other message; long-term maintenance debt
**Option C — Phased hybrid (recommended).**
Phase 1: hand-wire the snapshot payload (raw file bytes — trivially a byte stream, no per-entry
structure) to make Req 4 production-real. Phase 2: solve `Data` encoding properly in the typemap
for `WalCatchupEntry` (structured, batched, used on the hot path) and migrate the snapshot path to
it. Until both land, keep the spec phase below `implemented` or gate promotion on a data-plane
flag.
- ✅ De-risks incrementally; snapshot (rare, big) and WAL (frequent, small) have different shapes
- ❌ Two-step migration; temporary inconsistency between the two paths
### Gap 2 — Convergence threshold semantics (Req 2.3 vs code)
- **Option A**: change code to the requirement's majority formula (`ceil(n_peers/2) − 1` standbys).
Restores availability (one dead standby in a 3-node cluster no longer forces the 60s degraded
timeout) and matches Raft quorum; safe because safe_committed_lsn already derives from a majority.
- **Option B**: keep all-standbys convergence and amend requirements.md + tasks.md with the
rationale (strongest consistency at startup; degraded timeout covers minority loss).
- Either way, **reconcile the three formulas** (requirements 2.3, task 5.2, code) — they currently
disagree for any n ≥ 5.
### Gap 3 — Small correctness fixes (extend existing components)
1. Zero `standby_converged[]` / `standby_reported_*[]` in `raft_cb_become_leader` on COMMITTING
entry (Req 2.1 — re-election staleness).
2. `unlink(snap.part)` on standby reconnect/ctrlhello (Req 4.4 — stale partial file).
3. Create a fresh checkpoint (via delaymgr — `persist_checkpoint` is blocking I/O, must not run in
the vtask/heartbeat-reply context) when SEND_SNAPSHOT fires and the on-disk snapshot is older
than `oldest_wal_lsn` coverage requires (Req 4.1).
4. Consider caching `ha_compute_oldest_wal_lsn` — currently an opendir/readdir scan per heartbeat
reply per standby in vtask context (ha.c:2538-2543).
5. Optionally promote batch limits to `ha_config_t` (Req 5.2 "configurable").
## Effort and Risk
| Gap | Effort | Risk | Justification |
|---|---|---|---|
| 1 — Payload data plane (Option C) | **M–L (5–10d)** | **Medium** | Snapshot raw-bytes path is S; typemap `Data` codegen + batched WAL payload apply + ASAN-clean lifetime mgmt (current `(xce_data_t *)pbuf` cast at config_repl.c:1529 is already suspect) + E2E re-verification is the bulk |
| 2 — Threshold reconciliation | **S (0.5–1d)** | **Low** | One comparison + spec text alignment; existing tests pin the 3-node case |
| 3.1 — Tracking reset on COMMITTING | **S (<0.5d)** | **Low** | memset 3 arrays at ha.c:510; add re-election unit test |
| 3.2 — snap.part discard | **S (<0.5d)** | **Low** | unlink in handle_ctrlhello standby path |
| 3.3 — Fresh checkpoint on SEND_SNAPSHOT | **M (2–3d)** | **Medium** | Blocking checkpoint must hop to delaymgr; interaction with concurrent WAL writes |
| 3.4 — oldest_wal_lsn caching | **S (0.5d)** | **Low** | Cache + invalidate on segment GC |
| 3.5 — Configurable batch limits | **S (0.5d)** | **Low** | Plumb two fields through ha_config_t |
## Recommendations
1. **Do not mark this spec `implemented`.** All tasks are `[x]` but Requirements 3, 4, 5 are only
control-plane complete; the system currently reports convergence it has not achieved. Reopen
targeted tasks (suggest: 7.x series) for the payload data plane rather than re-flagging 2.x/3.5.
2. Land Gap 3.1 and 3.2 immediately (S, Low) — they are latent correctness bugs independent of the
payload work.
3. Decide Gap 2 (threshold) explicitly and record it in the Decision Log; today three documents
disagree.
4. Sequence the payload work after coordinating with **ha-config-bootstrap task 7.6**
(`repl_broadcast` TCP wiring) — the live path and catch-up path should share one entry
serialization, or the two specs will produce divergent wire formats for the same WAL entry.
5. Add an E2E assertion that verifies **entity state** (e.g., a submitted job visible on the
promoted standby), not just `applied_lsn` equality — the current tests 6.5–6.7 pass despite the
data-plane gap because they only observe LSN/log convergence.
## Research Needed
- How the typemap codegen represents capnp `Data` (`xce_data_t` tagged union table dispatch) and
what it takes to encode an arbitrary byte buffer — prerequisite for Gap 1 Option A/C phase 2.
- Whether convergence should additionally verify content (e.g., last-entry CRC chained via
`wal_entry_hdr_t.crc/prev_crc` in stateReport) so header-only application can never report
false convergence again.
- Ownership boundary with ha-config-bootstrap task 7.6: which spec owns live WAL entry
serialization, and can `send_wal_batch` be reused for it.
- Req 5.5 "durably written" on the standby: is in-memory entity restore + in-memory applied_lsn
acceptable (current documented design), or should batch apply fsync a standby-side WAL copy.

5
.kiro/specs/ha-repl-hardening/spec.json
View file

@ -1,7 +1,7 @@
{
"feature_name": "ha-repl-hardening",
"created_at": "2026-06-04T03:30:00Z",
"updated_at": "2026-06-04T10:35:00Z",
"updated_at": "2026-06-09T17:45:00Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
@ -18,5 +18,6 @@
"approved": true
}
},
"ready_for_implementation": true
"ready_for_implementation": true,
"note": "Reopened 2026-06-09: payload data plane missing (tasks 7.1-7.7); do not mark implemented until task series 7 lands"
}

46
.kiro/specs/ha-repl-hardening/tasks.md
View file

@ -254,3 +254,49 @@
- Verify promoted primary serves from committed state (no data loss)
- Full ASAN build: `meson test --suite controller` green
- _Requirements: 6.2, 6.3, 6.4_
---
## Reopened 2026-06-09 — Gap Analysis Validation Pass
The 2026-06-09 gap analysis (research.md) found the replication **data plane missing**:
snapshot chunks ship with `payload = NULL` (config_repl.c "encoding deferred"; cluster.c)
and WAL catch-up applies entries header-only (`payload_len=0`), so `apply_wal_entry`
advances LSN without restoring entities. Standbys report "converged" without holding the
state. Tests 6.5–6.7 pass because they assert LSN convergence only. The spec MUST NOT be
marked `implemented` until this series lands.
- [ ] 7. Replication payload data plane
- [ ] 7.1 Define the payload wire format: capnp `Data` field + typemap codegen for WAL
entries and snapshot chunks. Coordinate with ha-config-bootstrap task 7.6 — live
`repl_broadcast` is also a no-send stub and must share the same format.
- _Requirements: 3.1, 4.2, 5.3_
- _Boundary: Wire format (controller.capnp, config_repl.c)_
- [ ] 7.2 Snapshot chunk payload transmission (primary) + chunk write to `snap.part`
(standby); verify total size before atomic rename (phased option: hand-wire raw bytes
first if typemap codegen is blocked)
- _Requirements: 4.1, 4.2, 4.3_
- _Boundary: Snapshot transfer (config_repl.c, cluster.c)_
- [ ] 7.3 WAL catch-up entry payload transmission + entity restoration in
`apply_wal_entry` (e.g. `WAL_JOB_SUBMIT` recreates the job); `applied_lsn` advances
only after the entity is applied
- _Requirements: 3.1, 3.4, 5.3_
- _Boundary: WAL catch-up (ha.c, config_repl.c, persist.c)_
- [ ] 7.4 Re-zero convergence tracking arrays (`standby_reported_*`,
`standby_converged`) on entry to CTRL_PRIMARY_COMMITTING — stale convergence from a
prior term must not satisfy the gate after re-election
- _Requirements: 2.1_
- _Boundary: Convergence tracking (ha.c)_
- [ ] 7.5 Reconcile the convergence threshold: code requires ALL standbys (`n_peers−1`),
requirements say `ceil(n/2)−1`, tasks say `ceil(n/2)` — record the decision in the
Decision Log and align code + docs (matters for n ≥ 5)
- _Requirements: 2.4, 6.1_
- _Boundary: Quorum policy (ha.c, design.md)_
- [ ] 7.6 Production `unlink(snap.part)` on mid-transfer reconnect (Req 4.4) and fresh
checkpoint creation at snapshot start instead of reusing the old checkpoint (Req 4.1)
- _Requirements: 4.1, 4.4_
- _Boundary: Snapshot lifecycle (config_repl.c)_
- [ ] 7.7 E2E test: assert standby ENTITY state after convergence (submit job →
failover → promoted primary serves the job), not just LSN equality
- _Requirements: 6.2, 6.4_
- _Boundary: Integration test (deploy-ha)_

129
.kiro/specs/raft-library/research.md
View file

@ -50,3 +50,132 @@
- **Cross-spec integration**: `election-iomgr-refactor` uses `raft_start_election` instead of the old `am_i_primary_quorum`. The adapter callbacks in ha.c wire `on_send_vote_request` → iomgr async connect, `on_become_leader` → `cluster_init_phase2` + config push, `on_become_follower` → `CTRL_STANDBY`. The `on_vote_eligible` callback implements the freshness gate (config_tag + applied_lsn comparison) and role gate (`CTRL_PRIMARY_COMMITTING` → deny).
- **Test coverage**: 34+ Unity test cases across 5 groups (A–H) covering basic election correctness, voted_for invariant, eligibility policy, quorum evaluation, election timeout, RDHUP failover, and split-brain prevention. All pass under ASAN.
- **Recommendation**: The spec can be marked `implemented`.
---
# Gap Analysis — Validation Pass (2026-06-09)
> Scope: requirements.md (9 requirement groups, 31 acceptance criteria) vs. on-disk working tree
> (uncommitted changes to `src/lib/raft/raft.c`, `src/controller/ha.c`, `ha.h`, `myapp.c` analyzed as-is).
> spec.json phase: `tasks-generated`, all tasks.md checkboxes `[x]` except parent rollups 2/3/4/5/6.
## 1. Current State Summary
- Library exists and is build-integrated: `src/lib/raft/raft.h` (276 lines), `raft.c` (290 lines),
`raft_sim.c` (207 lines); wired in `src/lib/meson.build` (raft.c at line 108; test_raft.c + raft_sim.c
under the test executable at lines 175–176, simulator guarded by `XCE_TESTING`).
- Test suite: `src/lib/testsuite/test_raft.c` (2990 lines, 47 test functions, 79 RUN_TEST invocations)
covering unit tests, simulator tests, and protocol groups A–H plus F/G split-brain scenarios.
- Controller adapter live: `ha_state_t` embeds `raft_state_t raft` (`ha.h:183`); 9 callbacks implemented
in `ha.c` (`s_real_cbs` at ha.c:735); `raft_init` called in ha init (ha.c:854) and re-initialized with
real topology by `ha_raft_reinit` (ha.c:~2981–3008).
- Production dispatch fully delegated: vote requests → `raft_on_vote_request` (cluster.c:373),
vote replies → `raft_on_vote_reply` (ha.c:2361), RDHUP → `raft_on_leader_lost` (ha.c:3290),
election start → `raft_start_election` (ha.c:3146).
## 2. Requirement-to-Asset Map
| Req | Criterion | Asset(s) | Status |
|---|---|---|---|
| 1.1 | Per-node state (term/role/voted_for/n_peers) | `raft_state_t` (raft.h:23–38) | Implemented |
| 1.2 | Init defaults term=0/FOLLOWER/-1 | `raft_init` (raft.c:13–55); H2 test | Implemented (extended: restores `initial_term`/`initial_voted_for` from config) |
| 1.3 | No endpoint_t/SSL/iomgr/cluster_t/capnp in header | raft.h includes only errno/stdint/string (+stdio under XCE_TESTING) | Implemented |
| 2.1 | start_election: term++, self-vote, CANDIDATE, send RPCs | raft.c:57–104 | Implemented (also persists self-vote before RPCs — stronger than spec) |
| 2.2 | Single-node immediate LEADER | raft.c:83–88; tests A1, single_node | Implemented |
| 2.3 | -EALREADY if CANDIDATE/LEADER | raft.c:60–61; tests already_candidate/leader | Implemented |
| 3.1 | term > current: upgrade, clear vote, FOLLOWER, evaluate | raft.c:121–130; B2 | Implemented (extra: fires `on_become_follower` if was LEADER — split-brain fix, uncommitted change) |
| 3.2 | term < current: deny, return current term | raft.c:110–115; B3 | Implemented |
| 3.3 | One vote per term | raft.c:157–164; B1/B4/B5 | Implemented (extension: idempotent re-grant to the SAME candidate same term — beyond spec text; see Gap-3) |
| 3.4 | Freshness gate (config_tag/LSN) before grant | Split: caller-side gate in cluster.c:352–356 + `on_vote_eligible` hook (raft.c:167–173) | Partial — behavior present but located in the dispatcher, not in `on_vote_eligible` as tasks.md 6.2 states; `raft_cb_vote_eligible` (ha.c:202) intentionally skips freshness. Doc/code drift. |
| 3.5 | LEADER or COMMITTING deny ALL vote requests regardless of term | raft.c:135–148 (same-term deny only); COMMITTING gate in `on_vote_eligible` adapter | Partial (intentional deviation) — higher-term request now steps the leader down instead of denying, to prevent the COMMITTING-timeout split-brain. Requirement text is outdated. |
| 4.1 | Quorum (n/2)+1 → LEADER, on_become_leader once | raft.c:222–234; D1, A4 | Implemented |
| 4.2 | Reply term > current → FOLLOWER + on_become_follower | raft.c:197–206; D4 | Implemented |
| 4.3 | Reply after decided → no-op | raft.c:209–213; D3 | Implemented |
| 4.4 | All replied, no quorum → FOLLOWER | raft.c:237–244; D2 | Implemented |
| 5.1 | leader_lost: clear vote, FOLLOWER, delayed election (priority×timeout) | raft.c:268–276; F1 | Implemented |
| 5.2 | Skip election if voted for non-self during delay | `raft_check_and_start_election` raft.c:280–284; F2 | Implemented |
| 5.3 | Else start election internally | raft.c:286–289; F3 | Implemented |
| 6.1 | start_election schedules timeout | raft.c:101; E3 | Implemented |
| 6.2 | Timeout undecided → on_become_follower, cancel tracking | raft.c:260–265; E1 | Implemented |
| 6.3 | Timeout after decided → on_cancel_timeout no-op | raft.c:252–257; E2 | Implemented |
| 7.1 | raft_callbacks_t with 6 required fields | raft.h:66–96 (9 fields: 7 required + 2 optional) | Implemented (superset: adds `on_start_election_after_delay` as required, `on_vote_eligible`/`on_trace` optional) |
| 7.2 | void* ctx passed to every callback | raft_config_t.ctx → s->ctx, all call sites | Implemented |
| 7.3 | NULL required callback → error at init | raft.c:23–38 (-EINVAL); Hnull test | Implemented |
| 8.1 | Deterministic N-node simulator, synchronous routing | raft_sim.c + raft.h:197–274; sim tests | Implemented |
| 8.2 | Invariants: one-leader/3-node, dup deny, single-node, timeout re-election | A2/G1 (100 rounds), B1/B4, A1, E3 + G2 partition | Implemented |
| 8.3 | Compile + all tests pass under ASAN | tasks.md task 7 `[x]`; commits e10cf9a/361397a fixed ASAN issues | Unknown (stale) — last verified pre-dating current uncommitted edits to raft.c/ha.c; needs re-run |
| 9.1 | Adapter implements all callbacks via endpoint_t/iomgr | ha.c:107–744 (8 raft_cb_* + s_real_cbs) | Implemented |
| 9.2 | ha_start_election_async delegates; no direct ha->term/voted_for writes outside adapter | ha.c:3038–3161 delegates; only atomic loads + persist/restore remain | Partial — production path clean, but legacy `handle_request_vote()` (ha.c:1791–1873) still contains a full parallel vote-evaluation path with direct `ha->voted_for_term`/`voted_for_name` writes. It is production-dead (only `test_ha_vote_gate.c` calls it) but duplicates election state management contrary to the spirit of 9.2. |
| 9.3 | ha_phase2_hook / ha_rdhup_election_hook still functional | ha.c:495/631/2068/2147 (phase2), ha.c:3267–3284 (rdhup) | Implemented |
**Score: 27 of 31 acceptance criteria Implemented; 3 Partial (3.4, 3.5, 9.2 — all deliberate-deviation or residual-cleanup, not functional regressions); 1 Unknown (8.3 ASAN re-verification against the uncommitted working tree).**
### Constraints / behavior added beyond requirements
- **C1 (Constraint)**: `raft_init` rejects even `n_peers > 1` with -EINVAL (raft.c:21–22). Not in
requirements. A 2-node or 4-node HA deployment cannot initialize raft at all — verify this matches
deployment expectations (`deploy-ha.sh`, cluster config validation) or it becomes a runtime landmine.
- **C2 (Constraint)**: `voted_for_term` is unconditionally reset to 0 in `raft_init` (raft.c:43) even when
`initial_voted_for`/`initial_term` are restored from disk. After a restart, the duplicate-vote check
(`voted_for_term == term`, raft.c:157) cannot match the restored term, so a node that voted for X in
term T, restarted, then receives a request from Y in the same term T would fall through Step 4 and could
grant a second vote in T (subject to the caller-side freshness gate). Test H1 checks `term`/`voted_for`
but not `voted_for_term`. Potential one-vote-per-term violation across restart — needs confirmation.
## 3. Implementation Approach Options (for the remaining gaps)
### Option A: Extend existing components (close gaps in place) — RECOMMENDED
- raft.c: initialize `voted_for_term = (initial_voted_for >= 0) ? initial_term : 0` in `raft_init` (C2);
add an H6 restart-double-vote test.
- requirements.md: amend 3.5 (higher-term step-down is the intended behavior) and 3.4 (freshness gate
enforced by the dispatcher/callback split) so spec matches the as-built design; sync tasks.md 6.2 text.
- ha.c: delete or fold legacy `handle_request_vote()` into the raft path; retarget `test_ha_vote_gate.c`
at `raft_on_vote_request` + `on_vote_eligible` (closes 9.2).
- Re-run `/xce-build lib` + `/xce-build controller` under ASAN to refresh 8.3.
- Trade-offs: ✅ smallest diff, no architectural change, leverages existing test harness; ❌ touches a
legacy test file; requirements edit needs human re-approval per workflow.
### Option B: Create new components (formal vote-record module)
- Introduce a `raft_persisted_vote_t {term, voted_for}` pair persisted/restored atomically, replacing the
separate initial_term/initial_voted_for fields; new restart-focused test group.
- Trade-offs: ✅ makes the term-binding of a vote impossible to lose by construction; ❌ API churn in
raft_config_t and ha_raft_reinit for a problem fixable with one init line; over-engineering (YAGNI).
### Option C: Hybrid — fix C2 in place now (Option A), defer handle_request_vote removal to a follow-up
cleanup spec/task since it is production-dead.
- Trade-offs: ✅ de-risks the correctness fix from the test-refactor churn; ❌ leaves known duplication
in tree longer; two passes of review.
## 4. Effort & Risk
| Gap | Effort | Risk | Justification |
|---|---|---|---|
| C2 voted_for_term restore fix + test | S | Low | One-line init change + one Unity test in an established harness |
| Requirements/tasks doc sync (3.4, 3.5) | S | Low | Documentation only; needs approval round-trip |
| Legacy handle_request_vote removal (9.2) | S–M | Medium | Production-dead but test_ha_vote_gate.c must be retargeted; risk of disturbing controller test baseline |
| ASAN re-verification (8.3) | S | Low | Mechanical `/xce-build` run; fixes only if new findings |
| C1 even-n_peers policy confirmation | S | Medium | If 2-node deployments are expected, this is a hard init failure discovered only at runtime |
Overall remaining work: **S–M total, Low–Medium risk** — the library and integration are functionally complete.
## 5. Recommendations for Design/Closure Phase
1. Adopt **Option A**. Fix C2 (`voted_for_term` restore) first — it is the only candidate correctness
gap; everything else is documentation or hygiene.
2. Update requirements.md 3.5 and 3.4 to match the as-built (and better) behavior: leader steps down on
higher term; freshness gate enforced caller-side before `raft_on_vote_request` with COMMITTING/liveness
in `on_vote_eligible`. Re-approve, then run `/kiro-verify-completion raft-library`.
3. Remove or refactor legacy `handle_request_vote` (ha.c:1791) to eliminate the last duplicated election
state path (Req 9.2), retargeting `test_ha_vote_gate.c`.
4. Re-run task 7 ASAN validation against the current working tree before committing (the verified state
predates the uncommitted raft.c Step-3/Step-4 rewrite).
## Research Needed
- **R1**: Confirm intended cluster-size policy — are 2-node (even) HA clusters a supported deployment?
If yes, the even-n_peers -EINVAL in raft_init must be relaxed; if no, add config-load validation so the
failure surfaces at startup with a clear message rather than as a silent raft_init failure.
- **R2**: Verify whether the restart double-vote scenario (C2) is reachable in production timing — does
`ha_raft_reinit` run before the listener accepts vote requests? If unreachable, document why; fix anyway.
- **R3**: Confirm `raft_check_and_start_election` ignoring the `raft_start_election` return code
(raft.c:287–289) is safe in all adapter paths (comment claims -EALREADY unreachable; verify when an
inbound vote request flips role between RDHUP and the delayed check).

74
.kiro/specs/resval-id-decoupling/research.md
View file

@ -166,3 +166,77 @@ Ruled out: does not fix the design — just moves the source of truth without el
- **Task approval gap**: spec.json shows `tasks.approved: false`. All tasks are implemented but the tasks document was never formally approved. This is a process artifact — the implementation matches the approved design.
- **Cross-spec impact**: `scheduler-controller-msg-gap` extends the same scheduler handlers (`sched_init_sharedInst`, `make_sched_worker`) that were modified by this spec. The two specs are consistent — both use name-based resource resolution.
- **Recommendation**: Mark tasks as approved in spec.json, or run `/kiro-spec-status resval-id-decoupling` to sync.
---
# Gap Analysis — Validation Pass (2026-06-09)
## Method
Independent re-verification of every requirement (requirements.md R1–R6) against the current code on branch `persistent_ha`, via grep + direct read of the controller, scheduler, libxce, capnp schema, and Go API-server sources. Prior audit (2026-06-08) claimed ~100% completion; this pass confirms that claim and adds two residual findings it missed.
## Current State Summary
- The feature is **implemented end-to-end**. `collect_resval_with_idx` and `free_resval_with_idx_list` no longer exist anywhere in `src/` (zero grep matches). The replacement `collect_resval(var resmgr, zarray_t*, xlist_t*)` lives at `src/controller/dataex.c:23` and iterates the resmgr (names) rather than the zarray (slots), exactly as designed.
- Wire format: `src/schema/controller.capnp` — `Worker.resvals @5` (line 138) and `Conf4Sched.resvals @6` (line 239) are both `List(NameValue)`. Field ordinals preserved.
- All tasks in tasks.md are `[x]` (1.1–8.3), but `spec.json` still shows `tasks.approved: false` (phase: `tasks-generated`). Implementation landed despite the unapproved tasks document — a process artifact, not a code gap.
## Requirement-to-Asset Map
| Req | EARS | Asset(s) | Status |
|-----|------|----------|--------|
| R1 — name-keyed shared resvals in initial config | R1-1 | `controller.capnp:239` `List(NameValue)`; `src/controller/dataex.c:381` `collect_resval(c->resmgr, &c->sharedinst, l)`; `src/controller/sched.c:120` casts to `xce_name_val_t**` | **Implemented** |
| | R1-2 | `src/scheduler/src/scheduler.c:574-599` `sched_init_sharedInst`: `key_find(sched->resmgr, nv->name)` → `zarray_put(&sched->sharedInst, r->idx, val)` | **Implemented** |
| | R1-3 | `scheduler.c:587-592`: WARN + `continue` on unknown name; NULL name/entry also skipped | **Implemented** |
| | R1-4 | Same name→`r->idx` resolution as Path C handler (`scheduler.c:782-797`) | **Implemented** (by inspection; no equivalence test exists — see R6) |
| R2 — name-keyed per-worker resvals to scheduler | R2-1 | `controller.capnp:138` `List(NameValue)`; `dataex.c:151-166` `collect_resvals_workers(resmgr,...)`; `sched.c:442` | **Implemented** |
| | R2-2 | `src/scheduler/src/schedobj.c:274-292` `make_sched_worker`: name lookup → `zarray_put(&worker->resvals, r->idx, val)` | **Implemented** |
| | R2-3 | `schedobj.c:283-288`: WARN + skip, worker construction continues | **Implemented** |
| | R2-4 | Resolution happens at each receipt against the live `sched->resmgr`; no index is cached across messages | **Implemented** (by design; no automated test for resource add/remove mid-run) |
| R3 — name-keyed worker resvals to API server | R3-1 | `src/controller/apiserver.c:367-369`: same `collect_resval` + `xce_name_val_t**` path as the scheduler push | **Implemented** |
| | R3-2 | `src/frontend/apiserver/backend/convert.go:100-105` iterates `resvals_to_sched`, `convert.go:283` `cNameValToWorkerResval` (name-keyed, nil-checked); `convert_pure.go:210`; `models/hosts.go:94` `ResourceValues` | **Implemented** |
| R4 — Path C unchanged | R4-1 | `src/scheduler/src/handler.c:82-103` (`Data_data_resvals` → `XCE_SCHED_OP_UPDATE_RESVALS_FULL`) and `scheduler.c:782-800` untouched, still `xce_resval_t` name-keyed | **Implemented** |
| | R4-2 | `xce_conf4sched_t.resvals` (`typemap.h:62`) and `xce_worker_t.resvals_to_sched` (`types.h:219`) are `xce_name_val_t**`; codec auto-generated; `free_name_val_list` in `src/lib/utils.c:388` / `utils.h:130`; all controller free sites converted | **Implemented** |
| R5 — remove index-keyed call sites | R5-1 | Zero cross-subsystem index-keyed transfers remain (grep-verified); both receivers resolve by name | **Implemented** |
| | R5-2 | `xce_val_with_idx_t` typedef (`src/include/xce/types.h:38-41`) and capnp `struct ValueWithIdx` (`controller.capnp:176`) still exist with **zero users and no scoping comment** | **Partial** (SHOULD-level: dead type lingers, contrary to "removed or restricted ... with a comment") |
| R6 — no regression / correctness under divergent indices | R6-1 | Correct by code inspection (name resolution is order-independent); **no automated test** registers resources in different orders and asserts `sharedInst` contents | **Partial** (code Implemented, test Missing) |
| | R6-2 | Existing controller/scheduler suites passed at task 8.2; same-order behavior preserved | **Implemented** |
| | R6-3 | `calc_wanted_*` paths read `sharedInst`/`worker->resvals` via scheduler-side `r->idx` (scheduler testsuite exercises these at zarray level) | **Implemented** |
### Adjacent finding (Constraint — out of spec scope, same defect class)
`XCE_SCHED_OP_UPDATE_RESVALS` (per-host running update, `handler.c:231-251` → `scheduler.c:801-816`) receives name-keyed `xce_resval_t` records but stores them with a **hardcoded slot**: `zarray_put(&w->resvals, 0, rv->val)` — it finds the worker by `rv->host` but never resolves `rv->name` to `r->idx`. Any per-host update for a resource whose scheduler idx ≠ 0 lands in the wrong slot (or clobbers slot 0). This path was not enumerated in this spec (Paths A/B/C only) and belongs to the `scheduler-controller-msg-gap` surface, but it violates the spirit of EARS-R5-1. Flagged for follow-up, not a regression of this feature.
## Implementation Approach Options (for residual gaps)
### Option A — Minimal cleanup within this spec (Recommended)
Delete the dead `xce_val_with_idx_t` typedef and capnp `ValueWithIdx` struct (or add the scoping comment EARS-R5-2 asks for); add one scheduler testsuite case that feeds `sched_init_sharedInst`/`make_sched_worker` a name-val list against a resmgr populated in a different order and asserts slot contents (closes R6-1 test gap).
- ✅ Closes both Partial items with ~2 files + 1 test; no wire change (struct ordinals of NameValue untouched)
- ❌ Touches a shared header (`types.h`) — full rebuild needed to confirm nothing else references the type
### Option B — Fold residuals into `scheduler-controller-msg-gap`
Fix the `zarray_put(..., 0, ...)` slot bug together with the R5-2/R6-1 residuals in the adjacent spec, since it already owns the per-host update handlers.
- ✅ Single coherent change set for all per-host resval handling; this spec can be closed as-is
- ❌ R5-2/R6-1 residue stays attached to the wrong spec's history; risk of being dropped if that spec stalls
### Option C — Accept and close
Mark tasks approved, record R5-2 (SHOULD) and the missing R6-1 test as known accepted debt.
- ✅ Zero effort; all MUST-level acceptance criteria are demonstrably met
- ❌ Dead type invites accidental reuse; the divergent-order scenario — the entire point of the feature — remains untested by automation
## Effort & Risk
- **Effort: S** — residual work is a dead-type deletion plus one table-driven C test; the feature itself requires no further code.
- **Risk: Low** — all changes verified in place and grep-confirmed; the only Medium-risk item is the adjacent slot-0 bug, which is pre-existing and outside this spec.
## Recommendations
1. Approve tasks in `spec.json` (`tasks.approved: true`) to close the process gap — implementation matches the approved design.
2. Take Option A: remove (or comment-scope) `xce_val_with_idx_t` / `ValueWithIdx`, and add the divergent-registration-order test for `sched_init_sharedInst` / `make_sched_worker` (TDD: the test also documents EARS-R6-1 permanently).
3. File the `XCE_SCHED_OP_UPDATE_RESVALS` slot-0 defect against `scheduler-controller-msg-gap` (or a standalone fix) — it is the last index-style hazard in the resval surface.
## Research Needed
- **RN-1**: Confirm `XCE_SCHED_OP_UPDATE_RESVALS` (`scheduler.c:814`) is a live path (who sends `schedUpdateResvals` capnp field @8, and with which resources) before fixing the hardcoded slot 0 — it may be stub/dead code.
- **RN-2**: Verify no out-of-tree or generated consumer (e.g., `src/build/lib/controller.capnp.c`) still references `ValueWithIdx` codecs before deleting the schema struct; a clean `ninja -C src/build` after removal settles this.
- **RN-3**: EARS-R2-4 (resource add/remove mid-run) has no test harness today; determine whether the scheduler testsuite can drive `sched_handle_resources` followed by a worker push without a live controller.

6
.kiro/specs/resval-id-decoupling/spec.json
View file

@ -1,9 +1,9 @@
{
"feature_name": "resval-id-decoupling",
"description": "investigate a better solution to pass resource val from controller to scheduler/api server, the current solution has a very strong assumption that the idx/id of resource in controller and scheduler should keep the same, but this assumption may be too strong, the key function is collect_resval_with_idx() in src/controller/dataex.c",
"phase": "tasks-generated",
"phase": "implemented",
"created_at": "2026-05-12T06:52:00-07:00",
"updated_at": "2026-05-12T07:45:00-07:00",
"updated_at": "2026-06-09T17:45:00Z",
"language": "en",
"status": "active",
"approvals": {
@ -17,7 +17,7 @@
},
"tasks": {
"generated": true,
"approved": false
"approved": true
}
}
}

98
.kiro/specs/scheduler-controller-msg-gap/research.md
View file

@ -109,4 +109,100 @@ All 19 NotifyData event tags:
- **No remaining gaps**: All 8 requirements have verified implementation. The 6 previously stubbed notify/dataset handlers (`schedUpdateJob`, `schedUpdateQueue`, `schedUpdateResvals`, `schedJobRusages`, `Data_data_resources`, `Data_data_resvals`) are wired with full business logic. Dead code (`pushmsg` branches, `sched_handle_jobpush`, `sched_handle_workerpush`) removed. Symbolic logging via `xce_notify_event_name()` and `xce_sched_op_name()` is in place.
- **Optional gap**: Task 9.9 (verify WARN log when handler returns non-zero) is marked optional — requires log capture infrastructure not yet present in the test harness.
- **Cross-spec relationship**: `resval-id-decoupling` uses the same pattern (name-based resource lookup via `sched->resmgr`) in `sched_init_sharedInst`. The `XCE_SCHED_OP_UPDATE_RESVALS_FULL` handler added in this spec also resolves by name — consistency is maintained.
- **Recommendation**: The spec can be marked `implemented`. Task 9.9 can be closed as deferred.
- **Recommendation**: The spec can be marked `implemented`. Task 9.9 can be closed as deferred.
---
# Gap Analysis — Validation Pass (2026-06-09)
> Method: per `.claude/skills/kiro-validate-gap/rules/gap-analysis.md`. Every requirement re-verified against current source on `persistent_ha` (handler.c, scheduler.c, schedobj.c, plugin.c, helper.c, resobj.c, controller sched.c, testsuite/test.c). This pass **revises the 2026-06-08 audit's "~100% complete" conclusion**: handler wiring is complete, but four acceptance criteria are only partially met, two latent bugs exist in the dequeue paths, payload memory is leaked for the new op types, and the Task 9 test checkboxes substantially overstate actual coverage.
## 1. Current State Summary
- All 7 controller-emitted NotifyData events (`src/controller/sched.c`: schedStartJob, schedExitJob, schedFailJob, schedUpdateJob, schedJobRusages, schedUpdateResvals, schedUpdateQueue) have non-stub scheduler handlers in `src/scheduler/src/handler.c` (`sched_handle_notify`, lines 275–323). Unknown events return `-EINVAL`.
- Both previously-empty dataset cases (`Data_data_resources`, `Data_data_resvals`) are wired in `sched_handle_dataset` (handler.c:130–144) with ownership transfer (`dataset->nrows = 0; dataset->rows = NULL`).
- Dead `Request_body_pushmsg` branch and `sched_handle_jobpush`/`sched_handle_workerpush` are fully removed; the only remaining reference is a comment in testsuite/test.c:459. Unknown `body_which` returns `-EINVAL` (handler.c:349–351).
- The enqueue→dequeue two-phase pattern is followed: all new handlers enqueue `xce_sched_update_request_t` via `enq_pend_update()`; `sched_update()` (scheduler.c:727–845) processes the five new ops on the main thread.
- Note: no controller code currently **pushes** `Data_data_resources`/`Data_data_resvals` to the scheduler — `src/controller/sched.c` pushes only schedjobs/schedworkers; `src/controller/opget.c` serves these tables only as get-responses. The Req 1 handlers are forward-looking (harmless, but untriggered in production today).
## 2. Requirement-to-Asset Map
| Req / AC | Asset(s) | Status | Notes |
|---|---|---|---|
| 1 / UB-1.1 (resources add **or update**) | handler.c:58–79 `sched_handle_resources`; scheduler.c:770–781 `XCE_SCHED_OP_UPDATE_RESOURCES` | **Partial** | Dequeue calls `makeResource()`, but `makeResource_` (src/lib/resobj.c:108–140) returns NULL with `EEXIST` when the name already exists — **update semantics are missing**; re-pushed catalogs silently fail to modify existing resources. Return value is also ignored (no log), violating the error-handling steering ("never silently swallow errors"). |
| 1 / UB-1.2 (resvals → sharedInst by index) | handler.c:81–102; scheduler.c:782–799 `UPDATE_RESVALS_FULL` | Implemented | Resolves resource by name via `key_find(sched->resmgr,…)`, puts at `r->idx` — consistent with `sched_init_sharedInst` and the resval-id-decoupling spec. Unknown resources WARN-and-skip. |
| 1 / UB-1.3 (skip NULL rows) | scheduler.c:773–775, 786–788 | Implemented | `continue` on NULL entries in both loops. |
| 1 / UB-1.4 (-ENOMEM, free partial) | handler.c calloc guards | Implemented | Handlers return `-ENOMEM`; no partial allocations to free at enqueue level. |
| 2 / UB-2.1, UB-2.2 (dead code gone) | handler.c:325–362 | **Implemented** | `Request_body_pushmsg` case and both stub functions absent from source; `grep` finds no definitions anywhere in src/scheduler/. |
| 3 / UB-3.1 (update mutable fields: state, priority, queue, resreq) | scheduler.c:707–725 `sched_update_existing_job` | **Partial** | Only `job->state` is copied from the update payload. Priority, queue, and resource-request fields named in the requirement are not updated. `plugin_job_update()` is called. |
| 3 / UB-3.2 (-ENOENT unknown jobid) | scheduler.c:716–718 | Implemented | Wait — caveat: `sched_update_existing_job` exists but the `XCE_SCHED_OP_UPDATE_JOB` dequeue case routes to `sched_update_jobs()` (scheduler.c:754–756), which is the bulk create path. Whether a notify-driven single-job update reaches `sched_update_existing_job` needs verification (see Research Needed R-2). |
| 3 / UB-3.3 (-EINVAL NULL payload) | handler.c:180–182 | Implemented | |
| 4 / UB-4.1, 4.2, 4.3 (queue update) | schedobj.c:485–511 `sched_update_existing_queue`; scheduler.c:767–769 | Implemented | status/priority/accept_interval/sched_delay/resreq all updated; resreq freed + reallocated; `-ENOENT` on unknown queue; `-EINVAL` on NULL; `plugin_queue_update()` called. |
| 5 / UB-5.1 (per-worker resvals update) | scheduler.c:801–818 `UPDATE_RESVALS` | **Partial — bug** | `zarray_put(&w->resvals, 0, rv->val)` writes **index 0 unconditionally**. Worker resvals are indexed by resource idx (`zarray_put(&(worker->resvals), r->idx, val)`, schedobj.c:291) and read back via `zarray_get(&w->resvals, i)` in resource.c:161. The handler must resolve `rv->name` → `sched->resmgr` → `r->idx` and put at that index; as written, every update clobbers slot 0 regardless of resource. |
| 5 / UB-5.2 (skip unknown host) | scheduler.c:808–811 | Implemented | `sched_get_worker()` NULL → `continue`. |
| 5 / UB-5.3 (zero values → return 0) | handler.c:235–237 | Implemented | NULL or `count == 0` short-circuits before enqueue. |
| 6 / UB-6.1 (**store** rusage on job + plugin) | scheduler.c:819–831 `JOB_RUSAGE`; plugin.c:361 `plugin_job_rusage_update` | **Partial** | Plugin hook is implemented and called, but the rusage data is **not stored on the job object** as the requirement states — after the plugin call the payload is dropped (and leaked, see §3). |
| 6 / UB-6.2 (-ENOENT unknown jobid) | scheduler.c:822–824 | Implemented | |
| 6 / UB-6.3 (-EINVAL NULL rusage) | handler.c:257–259 | Implemented | |
| 7 / UB-7.1 (WARN with op + errno) | scheduler.c:838–841 | Implemented | `WARN("%s: op %s failed: %s (%d)", …, sched_op_name(req->op), strerror(-rc), rc)` after the switch. |
| 7 / UB-7.2 (unknown notify → -EINVAL) | handler.c:315–317 | Implemented (non-regression confirmed) | |
| 7 / UB-7.3 (unknown body_which → -EINVAL) | handler.c:349–351 | Implemented (non-regression confirmed) | |
| 8 / UB-8.1 (op name in req handler log) | handler.c:330 | Implemented | `TRACE("%s op = %s", __func__, xce_request_op_name(request))`. |
| 8 / UB-8.2 (dataset table name) | handler.c:109 | Implemented | `xce_dataset_table(dataset->table)`. |
| 8 / UB-8.3 (notify event name) | handler.c:279 | Implemented | `xce_notify_event_name(notify->event)`. |
| 8 / UB-8.4 (UNKNOWN(%d) fallback) | src/lib/helper.c:367–415 | Implemented | All 19 NotifyData tags covered + `snprintf(buf, …, "UNKNOWN(%d)", event)`. |
| 8 / UB-8.5 (symbolic op name in WARN) | scheduler.c:685–706 `sched_op_name` | Implemented — **deviation** | Implemented as **static `sched_op_name()` in scheduler.c**, not `xce_sched_op_name()` in `src/lib/helper.c`/`src/include/helper.h` as design Component 8 and tasks 1.1/1.3 (both `[x]`) state. The deviation is arguably correct (the enum is scheduler-internal; libxce cannot see `xce_sched_op_t`), but tasks.md and design.md no longer reflect reality. Uses a non-thread-safe `static char buf[32]` — acceptable since only the main thread calls it; same pattern as the helper. |
| 9 (tests) | src/scheduler/testsuite/test.c | **Partial — checkboxes overstate coverage** | See §4. |
**Memory-ownership constraint (design risk realized):** `sched_handle_notify` nullifies `notify->data.*` after successful enqueue (handler.c:297–315), transferring ownership from `free_Request()` to the update request — but `sched_pend_update_killer` (scheduler.c:311–321) has empty `break;` for all five new ops and the dequeue cases never free the payloads. Result: every `schedUpdateQueue` leaks an `xce_queue_t`, every `schedUpdateResvals` leaks the `xce_resval_list_t` (entries + name/host strings; only `rv->val` survives into the zarray — and on the index-0 bug path even that gets clobbered without a free), every `schedJobRusages` leaks the rusages struct, and every resources/resvals dataset leaks the row array and row structs (`makeInstance` copies; originals are never freed). The `UPDATE_JOB` path is fine (existing killer frees via `free_Job_ptr`). The design's High-severity risk row predicted exactly this; the mitigation was only half-applied (transfer done, release missing).
## 3. Identified Gaps (priority order)
1. **G-1 (bug, correctness): UPDATE_RESVALS writes worker resval index 0 unconditionally** — scheduler.c:813. Breaks scheduling decisions that read per-resource values. (Req 5)
2. **G-2 (bug, correctness): UPDATE_RESOURCES cannot update an existing resource** — `makeResource` → `EEXIST` → NULL, return ignored, no log. "Add or update" is only "add". (Req 1 / UB-1.1)
3. **G-3 (leak): killer/dequeue never free new-op payloads** despite ownership transfer (queue, resval_list, rusages, resource/resval rows). Slow per-message leak in a long-lived daemon. (Design risk #1)
4. **G-4 (requirement shortfall): job update copies only `state`** — priority/queue/resreq not applied. (Req 3 / UB-3.1)
5. **G-5 (requirement shortfall): rusage not stored on job object** — only the plugin hook sees it. Requires a rusage field on `xce_sched_job_t` (none exists today). (Req 6 / UB-6.1)
6. **G-6 (test debt): tasks 9.2, 9.3, 9.4, 9.6, 9.7 are checked `[x]` but absent from test.c.** The only new tests are: 9.1 notify-event-name (real), an enum compile-presence check, 9.5 reduced to a NULL-guard exercised against a **local stub** of `sched_update_existing_queue` defined inside test.c:25–34 (the real schedobj.c implementation is never linked/tested), and 9.8 reduced to `TEST_PASS()` with a comment. `xce_sched_op_name`/`sched_op_name` has no test (9.1 claims it does). None of the enqueue handlers, dequeue cases, or `sched_update_existing_job` are tested.
7. **G-7 (doc drift): design Component 8 / tasks 1.1, 1.3 say `xce_sched_op_name` lives in lib/helper** — it is a static in scheduler.c. Update spec docs or move the function.
## 4. Implementation Approach Options
**Option A — Extend in place (minimal correctness patch).** Fix G-1 (resolve `rv->name` → resmgr idx, mirroring schedobj.c:284–291), G-2 (on `EEXIST` find existing resource and mutate fields, or add an `updateResource()` beside `makeResource()` in resobj.c; log NULL returns), G-3 (add `free_*` calls in `sched_pend_update_killer` per op, reusing the deep-free helpers in src/lib/request.c), G-4 (copy priority/queue/resreq in `sched_update_existing_job`), and G-7 (doc edit). Defer G-5/G-6.
- ✅ Smallest diff, all in files already touched by this spec; restores correctness fast.
- ❌ Leaves UB-6.1 unmet and test checkboxes false; test debt persists.
**Option B — New test program + extracted ops module.** Create `src/scheduler/testsuite/test_msg_handlers.c` (links scheduler objects + generated capnp headers, which test.c cannot include — the stated reason 9.8 became a comment), and optionally extract the five dequeue case bodies into `src/scheduler/src/msgops.c` with non-static functions so they are unit-testable without including scheduler.c.
- ✅ Makes every acceptance criterion genuinely testable; honest Task 9; aligns with testing steering (tiered test_*.c programs already exist in other daemons).
- ❌ Larger diff; new file + meson.build change; refactor risk in scheduler.c.
**Option C — Hybrid (recommended).** Phase 1: Option A bug/leak fixes (G-1…G-4, G-7) with TDD where linkable. Phase 2: Option B test program covering handler enqueue + dequeue paths, replacing the stub-based 9.5 and the TEST_PASS 9.8 with real assertions; un-check or re-implement tasks 9.2–9.8 accordingly. Phase 3 (optional, needs decision): G-5 — add a rusages field to `xce_sched_job_t` or formally amend Req 6 to "deliver to plugins only".
- ✅ Correctness first, honesty in tracking, requirement closure staged.
- ❌ Two-phase coordination; Phase 3 needs a requirements decision.
## 5. Effort & Risk
| Work item | Effort | Risk | Justification |
|---|---|---|---|
| G-1 resvals index fix | S | Low | One-line lookup mirroring existing schedobj.c pattern; resmgr API already used two cases above. |
| G-2 resource update-on-EEXIST | S–M | Medium | Touching shared libxce resobj.c affects controller too; scheduler-side find-and-mutate is the safer S variant. |
| G-3 killer frees | M | Medium | Must trace ownership per type against free_* helpers in request.c; wrong free = double-free under ASAN, wrong no-op = leak persists. |
| G-4 job field updates | S | Low | Copy 3 more fields + string realloc, same shape as queue update. |
| G-5 store rusage on job | M | Medium | New field on `xce_sched_job_t` + lifetime/free in `kill_sched_job`; or amend requirement (zero code). |
| G-6 real test program | M | Medium | New testsuite binary + meson wiring; capnp header dependency is the known friction that derailed Task 9 the first time. |
| G-7 doc sync | S | Low | Edit design.md/tasks.md only. |
| **Total (Option C, Phases 1–2)** | **M (≈4–6 days)** | **Medium** | All within established patterns; risk concentrated in memory-ownership work under ASAN. |
## 6. Recommendations for Design/Implementation Follow-up
1. Treat G-1 and G-2 as **defects against the already-approved spec** — fix before marking the spec `implemented`; both silently corrupt or ignore controller state updates.
2. Adopt **Option C**: in-place fixes first (TDD per project rules — failing test in the new test program first where possible), then the dedicated `test_msg_handlers.c` to make Task 9 honest. Un-check tasks 9.2–9.8 until real tests exist, or annotate them with the stub caveat.
3. Resolve G-3 with the design's own mitigation table: request struct owns the payload after nullification ⇒ `sched_pend_update_killer` must deep-free each new op's payload (reuse `free_Queue_ptr`/`free_Resource_ptr`-style helpers from src/lib/request.c if exported; otherwise add small free helpers).
4. Decide G-5 explicitly: either add the job-side rusage storage Req 6 promises, or amend UB-6.1 — do not leave the requirement and code disagreeing.
5. Sync design.md Component 8 and tasks 1.1/1.3 with the actual `sched_op_name` location (G-7).
## 7. Research Needed
- **R-1:** Which `free_*` deep-free helpers in src/lib/request.c are exported and safe for `xce_queue_t`, `xce_resval_list_t`, `xce_job_rusages_t`, `xce_resource_t*[]` — confirm against ASAN before wiring into the killer.
- **R-2:** Confirm the notify-driven `schedUpdateJob` actually reaches `sched_update_existing_job()`: the dequeue `XCE_SCHED_OP_UPDATE_JOB` case calls `sched_update_jobs()` (bulk path); verify `sched_update_jobs` performs update-in-place for existing jobids (design Component 5 said it should) rather than duplicating jobs.
- **R-3:** Is there a planned controller-side emitter for `Data_data_resources`/`Data_data_resvals` pushes to the scheduler (e.g., post-failover config resync in the persistent_ha work)? If not, decide whether Req 1's handlers stay forward-looking or get an emitter spec.
- **R-4:** Whether `makeInstance(&resobj_type, r)` deep-copies all of `xce_resource_t` (name string included) — determines whether resource row payloads can be freed immediately after `makeResource` returns.

83
.kiro/specs/worker-join-refactor/research.md
View file

@ -204,3 +204,86 @@ Design phase is **already approved**. Proceed directly to implementation:
- **No remaining gaps**: `handle_worker_join` now has explicit `key_find` → branch → `init_worker`/`rejoin_worker` → `apply_join_state` flow visible at the call site. `BEFORE_EVENT`/`AFTER_EVENT` hooks fire at the same event points. `on_worker_join` called after successful join. Admin status bits (`LOCKEDU | CLOSED | HIDDEN | CLOSED_BY_ADMIN`) preserved in `rejoin_worker`.
- **Build verification**: Run `/xce-build controller` to confirm `test_worker_join.c` (7 test cases) pass.
- **Recommendation**: The spec can be marked `implemented`.
---
# Gap Analysis — Validation Pass (2026-06-09)
## Analysis Summary
- **Status**: Implementation is **complete and committed** (`93ed282 feat(worker-join-refactor): implement apply_join_state, init_worker, rejoin_worker`). Working tree is clean for all spec-owned files. All tasks 1.1–5 are `[x]` in tasks.md.
- **Code verification**: Every requirement was re-checked against the actual source on branch `persistent_ha`. `handle_worker_join` (src/controller/join.c:33–119) now exposes the explicit check → create-or-update → status/resource-update flow; `apply_join_state` / `init_worker` / `rejoin_worker` are implemented in src/controller/worker.c:279–397; `add_worker` is fully removed (no remaining references anywhere in src/).
- **Only open item**: Req 5.4 (full testsuite green) could not be re-confirmed in this pass — no build artifact exists in the worktree. One `/xce-build controller` run is the remaining verification step.
## Current State Investigation
| Asset | Location | Finding |
|-------|----------|---------|
| `handle_worker_join` | `src/controller/join.c:33` | Rewritten. Fail-fast `find_matched_hostconfig` (no lock) → BEFORE_EVENT hook → brief workermgr READ lock + `key_find` + `reference(w)` → branch (`-ENOENT`→`init_worker`, found→`rejoin_worker`, other error→reject) → `vget` for AFTER_EVENT hook → `xce_reply` outside any lock → `priv->u.idx` set → `on_worker_join`. Outer workermgr WRITE lock is gone — the latent double-lock deadlock is eliminated. |
| `apply_join_state` | `src/controller/worker.c:283–337` | Static (exported under `-DXCE_TESTING`, which meson.build:148 sets for the test binary). Clears `XCE_WORKER_UNAVAIL`, sets `w->ep`, applies hosttype/model `"!"`-guarded overrides, `add_hostattrs_resvals`, mxj/maxcpus `set_host_slots`, dispatch-window + thresholds replacement, bool-resource loop. Precondition (per-worker WRITE lock held by caller) documented in comment. |
| `init_worker` | `src/controller/worker.c:339–364` | Builds temp `xce_worker_t`, calls `makeWorker` (which owns its own workermgr WRITE lock), `-errno` on NULL with ERROR log, then per-worker WRITE lock + `apply_join_state`, returns `w->idx`. |
| `rejoin_worker` | `src/controller/worker.c:366–397` | Per-worker WRITE lock; saves `admin_status = status & XCE_WORKER_ADMIN_MASK` (LOCKEDU\|CLOSED\|HIDDEN\|CLOSED_BY_ADMIN, defined worker.c:275); dereference/makeString replacement of hosttype/hostmodel; copies cpufactor/maxcpus/maxmem/maxswap/maxtmp/ndisks; `apply_join_state`; restores admin bits; returns `w->idx`. |
| `worker.h` | `src/controller/include/worker.h:11–14` | `init_worker` and `rejoin_worker` exported with `xce_hostconfig_t*` param; `add_worker` declaration removed (only unrelated `add_worker_to_xlist` remains in cluster.h). |
| Tests | `src/controller/testsuite/test_worker_join.c` | 7 `RUN_TEST` cases (3 × apply_join_state, 2 × init_worker, 2 × rejoin_worker), registered in `controller_test.c:154` and built via `meson.build:103`. |
| Out-of-scope guard | `src/lib/wobj.c` | `makeWorker`/`override_worker` untouched by the refactor commit (wobj.c admin-mask logic at lines 85–105 predates it, commit febb4a6). Boundary respected. |
## Requirement-to-Asset Map
| Req | Summary | Asset (file:line) | Status |
|-----|---------|-------------------|--------|
| 1.1 | key_find before create-or-update | join.c:75–79 (READ lock + key_find + reference) | Implemented |
| 1.2 | Branch on lookup result | join.c:81–97 (error / NULL / found branches) | Implemented |
| 1.3 | Reject non-ENOENT key_find error | join.c:81–84 (`idx = rc`, reply.status set, returned) | Implemented |
| 2.1 | Create via makeWorker | worker.c:349 (`makeWorker(c->workermgr, &tmp)`) | Implemented |
| 2.2 | Log failure, set reply status, skip on_worker_join | worker.c:350–353 + join.c:87–89, 107–112 (`if (idx < 0) return idx` before on_worker_join) | Implemented |
| 2.3 | Assign priv->u.idx | join.c:115 (before `on_worker_join`) | Implemented |
| 3.1 | Update hostattrs on rejoin | worker.c:376–386 (string + numeric attr copy) | Implemented |
| 3.2 | Update ep on rejoin | worker.c:294 (`w->ep = ep` inside apply_join_state, called from rejoin) | Implemented |
| 3.3 | Preserve admin bits | worker.c:374, 390 (save mask / restore after apply_join_state) | Implemented |
| 4.1 | Clear XCE_WORKER_UNAVAIL | worker.c:293 | Implemented |
| 4.2 | Apply hconfig overrides (type/model/mxj/window/thresholds/bool res) | worker.c:296–336 | Implemented |
| 4.3 | Host-attr resource values | worker.c:308 (`add_hostattrs_resvals`) | Implemented |
| 4.4 | Reject when hostconfig NULL, workermgr untouched | join.c:60–68 (fail-fast before BEFORE_EVENT hook and before any lock) | Implemented |
| 5.1 | BEFORE/AFTER hook order at same event points | join.c:71–72 (BEFORE, pre-key_find) and join.c:103–104 (AFTER, post-update, w via vget) | Implemented |
| 5.2 | on_worker_join after successful join (both paths) | join.c:116 (single post-branch call site) | Implemented |
| 5.3 | Critical section no longer than before | Locking narrowed vs. pre-refactor: brief READ for lookup, makeWorker-internal WRITE for create, per-worker WRITE for mutation; xce_reply outside all locks | Implemented |
| 5.4 | Existing testsuite passes unmodified | test_worker_join.c registered; no build dir in worktree to confirm a green run in this pass | Unknown — verify with `/xce-build controller` |
**Constraints noted (no action required)**:
- `apply_join_state` visibility is toggled by `XCE_TESTING` (worker.c:282–288) — tests link against the non-static symbol. This is the established testsuite pattern; keep the meson `-DXCE_TESTING` arg in sync.
- TOCTOU window between READ-lock `key_find` and `makeWorker` remains by design; `makeWorker_` degrades to `override_worker` (admin-bit-preserving since febb4a6), so behavior stays correct.
**Minor observation (LOW, optional)**: in the join.c error branch (rc < 0 && rc != -ENOENT), if `key_find` ever returned an error while also setting `w`, the `reference(w)` taken at join.c:77 would not be released. Current `key_find` semantics make this unreachable (`w` stays NULL on error), so this is a defensive-coding nit only.
## Implementation Approach Options
### Option A: Close out the spec as-is (Recommended)
Verify req 5.4 with one `/xce-build controller` run; if green, mark the spec `implemented` in spec.json and archive.
- ✅ Zero code change; everything else is verified at source level
- ✅ Matches the prior audit conclusion (2026-06-08) — now re-confirmed line-by-line
- ❌ None
### Option B: Add the defensive dereference in the key_find error branch before closing
One-line hardening (`if (w) { dereference(w); w = NULL; }` in the error branch) plus rebuild.
- ✅ Removes the theoretical reference leak
- ❌ Touches a committed, behavior-frozen handler for an unreachable path; out of the spec's acceptance criteria
### Option C: Fold verification into the next controller-wide build
Skip a dedicated build; rely on the next `/xce-build` from any active spec (e.g., HA work on this branch) to exercise `test_worker_join`.
- ✅ No extra build cycle
- ❌ Spec closure blocked on unrelated work; a regression would be attributed late
## Effort & Risk
- **Effort: S** — remaining work is one build/test run (Option A) or that plus a one-line nit (Option B); no design or implementation gaps remain.
- **Risk: Low** — pure verification of already-committed code following established patterns; all 16 functional criteria confirmed at source level; only the test-run evidence is missing.
## Recommendations
1. Run `/xce-build controller` to confirm the 7 `test_worker_join` cases and the rest of the controller testsuite pass (closes Req 5.4).
2. On green, update `spec.json` phase to `implemented` (or run `/kiro-verify-completion worker-join-refactor`).
3. Optionally apply the Option B one-line defensive dereference if the file is touched again for other reasons; do not reopen the spec for it.
## Research Needed
- None for this spec. Single carry-forward verification item: green `/xce-build controller` run on current HEAD (worker.c/join.c unmodified since 93ed282, but HA-branch changes to ha.c/raft.c in the working tree share the controller test binary — confirm no link/test interference).

4
.kiro/specs/worker-join-refactor/spec.json
View file

@ -1,9 +1,9 @@
{
"feature_name": "worker-join-refactor",
"created_at": "2026-05-15T00:00:00Z",
"updated_at": "2026-05-15T03:20:00Z",
"updated_at": "2026-06-09T17:45:00Z",
"language": "en",
"phase": "tasks-generated",
"phase": "implemented",
"approvals": {
"requirements": {
"generated": true,

299
.wolf/anatomy.md
View file

@ -1,30 +1,24 @@
# anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-06-09T08:36:14.668Z
> Files: 590 tracked | Anatomy hits: 0 | Misses: 0
## ../../../../../claude/.claude/plans/
- `functional-tumbling-squirrel.md` — Plan: Address ha-repl-hardening Design Review Issues (~924 tok)
- `hazy-exploring-swan.md` — Plan: HA Replication Hardening (~1100 tok)
- `prancy-herding-nest.md` — Plan: HA Election + Config Bootstrap Enhancement (~13241 tok)
## ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/
- `MEMORY.md` — Memory Index (~41 tok)
- `project_ha_bootstrap_progress.md` (~363 tok)
> Auto-maintained by OpenWolf. Last scanned: 2026-06-10T01:00:00.579Z
> Files: 505 tracked | Anatomy hits: 0 | Misses: 0
## ./
- `.clang-format` (~226 tok)
- `.gitignore` — Git ignore rules (~88 tok)
- `CLAUDE.md` — OpenWolf (~2900 tok)
- `.opencode.json` (~55 tok)
- `CLAUDE.md` — OpenWolf (~2137 tok)
- `package-lock.json` — npm lock file (~1039 tok)
- `package.json` — Node.js package manifest (~1 tok)
- `README.md` — Project documentation (~612 tok)
## .claude/
- `settings.json` (~441 tok)
- `settings.local.json` (~30 tok)
- `settings.json` (~497 tok)
- `settings.json.bak` (~329 tok)
- `settings.local.json` (~137 tok)
- `ucai-formatter-cache.local.json` (~17 tok)
## .claude/rules/
@ -183,11 +177,18 @@
## .claude/skills/xce-deploy-ha/
- `SKILL.md` — XCE HA Cluster Deployment (~1286 tok)
- `SKILL.md` — XCE HA Cluster Deployment (~1334 tok)
## .kiro/settings/
## .code-review-graph/
- `mcp.json` (~8 tok)
- `.gitignore` — Git ignore rules (~38 tok)
## .codegraph/
- `.gitignore` — Git ignore rules (~47 tok)
- `codegraph.db-shm` (~8738 tok)
- `daemon.log` (~1092 tok)
- `daemon.pid` (~44 tok)
## .kiro/settings/rules/
@ -229,11 +230,15 @@
- `structure.md` — Project Structure (~218 tok)
- `tech.md` — Technology Stack (~198 tok)
## .kiro/specs/.claude/
- `ucai-formatter-cache.local.json` (~17 tok)
## .kiro/specs/election-iomgr-refactor/
- `design.md` — Design Document: election-iomgr-refactor (~6671 tok)
- `requirements.md` — Requirements Document (~2695 tok)
- `research.md` — Research & Design Decisions (~5189 tok)
- `research.md` — Research & Design Decisions (~8796 tok)
- `spec.json` (~129 tok)
- `tasks.md` — Implementation Plan (~3693 tok)
@ -242,9 +247,9 @@
- `brief.md` — Brief: ha-election-raft-config (~16310 tok)
- `design.md` — Design: HA Config Bootstrap & Raft-Style Quorum Election (~11439 tok)
- `election-trace.md` — Election Trace: 3-Node Cold Start (2026-06-04 00:15:58) (~2907 tok)
- `manual-tests.md` — Manual Integration Tests: ha-config-bootstrap (~3677 tok)
- `manual-tests.md` — Manual Integration Tests: ha-config-bootstrap (~3748 tok)
- `requirements.md` — Requirements: HA Config Bootstrap & Raft-Style Quorum Election (~3890 tok)
- `research.md` — Cross-Spec Gap Audit (2026-06-03) (~8392 tok)
- `research.md` — Cross-Spec Gap Audit (2026-06-03) (~14349 tok)
- `spec.json` (~198 tok)
- `tasks.md` — Implementation Plan: HA Config Bootstrap & Raft-Style Quorum Election (~5619 tok)
@ -253,7 +258,7 @@
- `design.md` — Design Document: ha-heartbeat-liveness (~5409 tok)
- `manual-tests.md` — Manual Integration Tests: ha-heartbeat-liveness — Task 4.5 (~1726 tok)
- `requirements.md` — Requirements Document (~2054 tok)
- `research.md` — Gap Analysis: ha-heartbeat-liveness (~2290 tok)
- `research.md` — Gap Analysis: ha-heartbeat-liveness (~5437 tok)
- `spec.json` (~129 tok)
- `tasks.md` — Implementation Plan (~2227 tok)
@ -261,7 +266,7 @@
- `design.md` — Design Document: ha-peer-connection-reuse (~5369 tok)
- `requirements.md` — Requirements Document (~1605 tok)
- `research.md` — Research & Design Decisions: ha-peer-connection-reuse (~3801 tok)
- `research.md` — Research & Design Decisions: ha-peer-connection-reuse (~6598 tok)
- `spec.json` (~130 tok)
- `tasks.md` — Implementation Plan: ha-peer-connection-reuse (~2022 tok)
@ -270,15 +275,15 @@
- `design.md` — Design: HA Replication Hardening (~5514 tok)
- `raft-tla-audit.md` — Raft TLA+ Specification Audit (~1591 tok)
- `requirements.md` — Requirements Document (~2817 tok)
- `research.md` — Gap Analysis: HA Replication Hardening (~2756 tok)
- `spec.json` (~128 tok)
- `tasks.md` — Implementation Plan: HA Replication Hardening (~3816 tok)
- `research.md` — Gap Analysis: HA Replication Hardening (~6234 tok)
- `spec.json` (~165 tok)
- `tasks.md` — Implementation Plan: HA Replication Hardening (~4464 tok)
## .kiro/specs/raft-library/
- `design.md` — Design Document: raft-library (~7756 tok)
- `requirements.md` — Requirements Document (~2115 tok)
- `research.md` — Research & Design Decisions: raft-library (~949 tok)
- `research.md` — Research & Design Decisions: raft-library (~3768 tok)
- `spec.json` (~127 tok)
- `tasks.md` — Implementation Plan (~4312 tok)
@ -286,15 +291,15 @@
- `design.md` — Design: resval-id-decoupling (~3914 tok)
- `requirements.md` — Requirements: resval-id-decoupling (~2360 tok)
- `research.md` — Gap Analysis: resval-id-decoupling (~3505 tok)
- `spec.json` — Declares is (~229 tok)
- `research.md` — Gap Analysis: resval-id-decoupling (~5568 tok)
- `spec.json` — Declares is (~226 tok)
- `tasks.md` — Implementation Tasks: resval-id-decoupling (~2424 tok)
## .kiro/specs/scheduler-controller-msg-gap/
- `design.md` — Design: scheduler-controller-msg-gap (~4756 tok)
- `requirements.md` — Requirements: scheduler-controller-msg-gap (~2712 tok)
- `research.md` — Research Log: scheduler-controller-msg-gap (~1854 tok)
- `research.md` — Research Log: scheduler-controller-msg-gap (~5630 tok)
- `spec.json` (~184 tok)
- `tasks.md` — Tasks: scheduler-controller-msg-gap (~2324 tok)
@ -302,8 +307,8 @@
- `design.md` — Design Document: worker-join-refactor (~4709 tok)
- `requirements.md` — Requirements Document (~1401 tok)
- `research.md` — Research Log: worker-join-refactor (~3357 tok)
- `spec.json` (~129 tok)
- `research.md` — Research Log: worker-join-refactor (~5421 tok)
- `spec.json` (~128 tok)
- `tasks.md` — Implementation Plan (~1658 tok)
## .kiro/steering/
@ -326,6 +331,15 @@
- `2026-05-13-config-replication.md` — Config Replication Implementation Plan (~21611 tok)
- `2026-05-20-config-repl-wire-protocol.md` — Config Replication — Wire Protocol Completion + Bug Fixes (~5134 tok)
## ha/data/
- `xce.crt` — ---BEGIN CERTIFICATE----- (~381 tok)
- `xce.key` — ---BEGIN PRIVATE KEY----- (~455 tok)
## raft/
- `raft.tla` — ------------------------------- MODULE raft --------------------------------- (~5350 tok)
## src/
- `flake.nix` (~571 tok)
@ -447,50 +461,47 @@
- `requestMachines.sh` (~20 tok)
- `requestReturnMachines.sh` (~38 tok)
## src/build/lib/
- `controller.capnp.c` — include "controller.capnp.h" (~134780 tok)
## src/controller/
- `.ha.c.~undo-tree~` — Declares xce_cluster_config_t (~40575 tok)
- `apiserver.c` — include <errno.h> (~5303 tok)
- `CLAUDE.md` — XCE Controller (~2625 tok)
- `cluster.c` — include "config.h" (~12219 tok)
- `config_repl.c` — define _GNU_SOURCE (~14728 tok)
- `config_repl.h` — pragma once (~2481 tok)
- `CLAUDE.md` — XCE Controller (~2630 tok)
- `cluster.c` — include "config.h" (~13092 tok)
- `config_repl.c` — define _GNU_SOURCE (~15780 tok)
- `config_repl.h` — pragma once (~2658 tok)
- `ctrl_metrics.c` — include "ctrl_metrics.h" (~767 tok)
- `dataex.c` — include <errno.h> (~2466 tok)
- `ha.c` — define _GNU_SOURCE (~39651 tok)
- `ha.h` — ifndef HA_H (~9764 tok)
- `ha.c` — define _GNU_SOURCE (~42722 tok)
- `ha.h` — ifndef HA_H (~10718 tok)
- `job.c` — include <errno.h> (~8259 tok)
- `join.c` — include "config.h" (~2347 tok)
- `main.c` — include <argp.h> (~1996 tok)
- `join.c` — include "config.h" (~2514 tok)
- `main.c` — include <argp.h> (~2262 tok)
- `meson.build` (~1080 tok)
- `misc.c` — include <errno.h> (~1359 tok)
- `myapp.c` — define _GNU_SOURCE (~3209 tok)
- `myconf.c` — define _GNU_SOURCE (~1100 tok)
- `myapp.c` — define _GNU_SOURCE (~3831 tok)
- `myconf.c` — define _GNU_SOURCE (~1179 tok)
- `opget.c` — include "config.h" (~1725 tok)
- `opupd.c` — include "config.h" (~714 tok)
- `persist.c` — define _GNU_SOURCE (~13088 tok)
- `persist.c` — define _GNU_SOURCE (~14023 tok)
- `queue.c` — include <errno.h> (~1260 tok)
- `repl.c` — include <assert.h> (~3964 tok)
- `repl.c` — include <assert.h> (~4248 tok)
- `reply.c` — include "config.h" (~590 tok)
- `resources.c` — include <errno.h> (~50 tok)
- `sched.c` — include <errno.h> (~5230 tok)
- `startup_validation.c` — define _GNU_SOURCE (~1423 tok)
- `startup_validation.c` — define _GNU_SOURCE (~1524 tok)
- `task.c` — include <errno.h> (~1376 tok)
- `worker.c` — include <errno.h> (~5201 tok)
## src/controller/include/
- `cluster.h` — if !defined(_CLUSTER_H_) (~1669 tok)
- `cluster.h` — if !defined(_CLUSTER_H_) (~1788 tok)
- `worker.h` — if !defined(_WORKER_H_) (~190 tok)
## src/controller/testsuite/
- `controller_test.c` — include <argp.h> (~1305 tok)
- `controller_test.c` — include <argp.h> (~1398 tok)
- `test_cluster_persist.c` — define _GNU_SOURCE (~1062 tok)
- `test_config_repl.c` — define _GNU_SOURCE (~22128 tok)
- `test_config_repl.c` — define _GNU_SOURCE (~23709 tok)
- `test_ctrl_metrics.c` — define _GNU_SOURCE (~940 tok)
- `test_e2e_capnp.c` — include <stdint.h> (~1690 tok)
- `test_e2e_checkpoint.c` — define _GNU_SOURCE (~1793 tok)
@ -498,44 +509,41 @@
- `test_e2e_hooks.c` — define _GNU_SOURCE (~2346 tok)
- `test_e2e_wal.c` — define _GNU_SOURCE (~2546 tok)
- `test_gc_retention.c` — define _GNU_SOURCE (~2447 tok)
- `test_ha_bootstrap_support.c` — ifdef XCE_TESTING (~1008 tok)
- `test_ha_bootstrap.c` — does: defer_config_fetch (~5828 tok)
- `test_ha_bootstrap_support.c` — ifdef XCE_TESTING (~1080 tok)
- `test_ha_bootstrap.c` — does: defer_config_fetch (~6244 tok)
- `test_ha_cluster.c` — include <stdatomic.h> (~1351 tok)
- `test_ha_connect.c` — define _GNU_SOURCE (~1882 tok)
- `test_ha_connection_reuse.c` — define _GNU_SOURCE (~6304 tok)
- `test_ha_ctrlhello.c` (~1322 tok)
- `test_ha_durability.c` — Declares char (~4075 tok)
- `test_ha_election.c` — Declares char (~11144 tok)
- `test_ha_evaluate_committing.c` — Declares checks (~7656 tok)
- `test_ha_failover.c` — define _GNU_SOURCE (~3781 tok)
- `test_ha_liveness_sigstop.sh` — test_ha_liveness_sigstop.sh — Task 4.5 integration test: missed-heartbeat (~2606 tok)
- `test_ha_liveness.c` — Declares directly (~6006 tok)
- `test_ha_rdhup_integration.c` — Declares char (~4347 tok)
- `test_ha_connection_reuse.c` — define _GNU_SOURCE (~6754 tok)
- `test_ha_ctrlhello.c` (~1416 tok)
- `test_ha_durability.c` — Declares char (~4366 tok)
- `test_ha_election.c` — Declares char (~11940 tok)
- `test_ha_evaluate_committing.c` — Declares checks (~8203 tok)
- `test_ha_failover.c` — define _GNU_SOURCE (~4051 tok)
- `test_ha_liveness.c` — Declares directly (~6435 tok)
- `test_ha_rdhup_integration.c` — Declares char (~4658 tok)
- `test_ha_redirect.c` — include <stdatomic.h> (~905 tok)
- `test_ha_repl_ack.c` (~3093 tok)
- `test_ha_repl_push.c` — Declares is (~2818 tok)
- `test_ha_role.c` — include <signal.h> (~9651 tok)
- `test_ha_start_election_async.c` — Declares returns (~1957 tok)
- `test_ha_startup.c` — Declares char (~8594 tok)
- `test_ha_vote_gate.c` — define _GNU_SOURCE (~2679 tok)
- `test_ha_vote_send.c` — define _GNU_SOURCE (~5635 tok)
- `test_ha_vote_send.c` — define _GNU_SOURCE (~3212 tok)
- `test_ha_repl_ack.c` (~3314 tok)
- `test_ha_repl_push.c` — Declares is (~3020 tok)
- `test_ha_role.c` — include <signal.h> (~10340 tok)
- `test_ha_start_election_async.c` — Declares returns (~2097 tok)
- `test_ha_startup.c` — Declares char (~9207 tok)
- `test_ha_vote_gate.c` — define _GNU_SOURCE (~2934 tok)
- `test_ha_vote_send.c` — define _GNU_SOURCE (~6038 tok)
- `test_hooks.c` — include <stdio.h> (~365 tok)
- `test_p1_integration.c` — include <stdatomic.h> (~1496 tok)
- `test_p1_integration.c` — include <stdatomic.h> (~1270 tok)
- `test_p2_integration.c` — include <errno.h> (~2096 tok)
- `test_p3_hardening.c` — define _GNU_SOURCE (~5279 tok)
- `test_p3_hardening.c` — define _GNU_SOURCE (~5003 tok)
- `test_p3_manual_failover.c` — include <stdatomic.h> (~984 tok)
- `test_persist_checkpoint.c` — define _GNU_SOURCE (~1815 tok)
- `test_persist_hooks.c` — define _GNU_SOURCE (~4222 tok)
- `test_persist_recovery.c` — define _GNU_SOURCE (~2290 tok)
- `test_persist_repl.c` — include <errno.h> (~1396 tok)
- `test_persist_schema.c` — include <stdint.h> (~1190 tok)
- `test_persist_truncate.c` — define _GNU_SOURCE (~1973 tok)
- `test_persist_truncate.c` — define _GNU_SOURCE (~2114 tok)
- `test_persist_types.c` — include <stddef.h> (~697 tok)
- `test_persist_wal.c` — define _GNU_SOURCE (~1708 tok)
- `test_repl_apply.c` — include <stdio.h> (~1862 tok)
- `test_repl_catchup.c` — include <errno.h> (~1183 tok)
- `test_repl_majority.c` — include <errno.h> (~2108 tok)
- `test_repl_majority.c` — include <errno.h> (~2258 tok)
- `test_repl_unit.c` — include <assert.h> (~1662 tok)
- `test_worker_join.c` — include <rtti.h> (~4851 tok)
- `test.c` — include <argp.h> (~18744 tok)
@ -580,7 +588,7 @@
- `control_host_test.go` — go:build !cgo (~211 tok)
- `convert_pure.go` — rawStartInfo (93 fields) (~3096 tok)
- `convert_test.go` — TestRawJobToModel_BasicFields, TestRawJobToModel_NilSubmit, TestRawJobToModel_Timestamps, TestRawWorkerToModel_BasicFields + 14 more (~6770 tok)
- `convert.go` — go:build cgo (~3126 tok)
- `convert.go` — go:build cgo (~3349 tok)
- `handler_test.go` — TestOnHostConfigUpdated_UnregisteredHostIgnored, TestOnHostConfigUpdated_DefaultPseudoHostIgnored, TestOnHostConfigUpdated_PartialMerge, TestOnClus... (~2994 tok)
- `handler.go` — DefaultHandler (8 fields); methods: OnJobCreated, OnJobUpdated, OnJobDeleted, OnJobFinished (~1054 tok)
- `heartbeat_test.go` — TestSetHeartbeatInterval_ValidValue, TestSetHeartbeatInterval_ZeroIgnored, TestSetHeartbeatInterval_NegativeIgnored, TestRunHeartbeat_CallsFunction... (~593 tok)
@ -856,138 +864,3 @@
- `config.go` — APIServerInfo (64 fields); methods: Len, Swap, Less, GetConfig (~1607 tok)
- `logger.go` — LoggerConfig (48 fields) (~1877 tok)
- `logger.yaml` (~205 tok)
- `schema.go` — ValidateYamlFormat, ValidateYAML (~753 tok)
## src/frontend/xcecli/config/schema/
- `logger_schema.json` (~930 tok)
## src/frontend/xcecli/filters/
- `host_filter.go` — HostsFilter (7 fields); methods: HandleHostFilter (~166 tok)
- `job_filter.go` — JobsFilter (61 fields); methods: HandleJobFilter (~1511 tok)
- `queue_filter.go` — QueuesFilter (7 fields); methods: HandleQueueFilter (~119 tok)
- `resource_filter.go` — ResourceDistFilter (13 fields); methods: HandleResourceDistFilter, HandleResourceDefFilter (~271 tok)
- `role_filter.go` — RolesFilter (9 fields); methods: HandleRoleFilter (~149 tok)
- `user_filter.go` — UserRolesFilter (8 fields); methods: HandleUserRoleFilter (~149 tok)
## src/frontend/xcecli/models/
- `cluster.go` — ClusterResponse (15 fields) (~201 tok)
- `exitstatus_test.go` — TestQueryJob_ExitStatus_Null, TestQueryJob_ExitStatus_String, TestQueryJob_ExitStatus_Zero (~292 tok)
- `hosts.go` — HostsResponse (56 fields); methods: UnmarshalJSON (~1288 tok)
- `jobs.go` — JobsResponse (149 fields); methods: UnmarshalJSON, String, UnmarshalJSON (~5415 tok)
- `logon.go` — LogonData define the data structure in the logon response (~168 tok)
- `ordermap.go` — UnmarshalOrderMapRow extracts the Data sub-object from the server's OrderMap wire format (~166 tok)
- `permissions.go` — PermsResponse (6 fields) (~98 tok)
- `queues.go` — QueuesResponse (33 fields); methods: UnmarshalJSON (~1072 tok)
- `resource.go` — ResourceValue (66 fields); methods: UnmarshalJSON, String, UnmarshalJSON, String (~1674 tok)
- `response_test.go` — TestHostsResponse_FullUnmarshal, TestQueuesResponse_FullUnmarshal, TestJobsResponse_FullUnmarshal, TestRolesResponse_FullUnmarshal + 3 more (~1644 tok)
- `roles.go` — CreateRoleRequest (25 fields); methods: UnmarshalJSON (~361 tok)
- `users.go` — AddRoleForUserRequest (26 fields); methods: UnmarshalJSON (~343 tok)
## src/frontend/xcecli/router/
- `router.go` — Execute (~919 tok)
## src/frontend/xcecli/test/
- `t.sh` (~118 tok)
## src/include/
- `attrs.h` — if !defined(_ATTRS_H_) (~26 tok)
- `bitset.h` — if !defined(_BITSET_H_) (~71 tok)
- `byteorder.h` — pragma once (~247 tok)
- `client.h` — if !defined(_CLIENT_H_) (~1023 tok)
- `clusterconfig.h` — if !defined(_CLUSTERCONFIG_H_) (~474 tok)
- `config.h` — if !defined(_CONFIG_H_) (~993 tok)
- `configobj.h` — if !defined(_CONFIGOBJ_H_) (~46 tok)
- `ctrl_metrics.h` — pragma once (~539 tok)
- `dataq.h` — if !defined(_DATAQ_H_) (~190 tok)
- `delay.h` — public delay API; adds delay_task_ms() and nextrun_ts field to delay_task_t (~320 tok)
- `group.h` — if !defined(_GROUP_H_) (~227 tok)
- `helper.h` — if !defined(_HELPER_H_) (~207 tok)
- `hook.h` — if !defined(__HOOK_H__) (~195 tok)
- `htab.h` — if !defined(_HTAB_H_) (~268 tok)
- `idgen.h` — if !defined(_IDGEN_H_) (~117 tok)
- `jobj.h` — if !defined(_JOBJ_H_) (~103 tok)
- `kv.h` — if !defined(_KV_H_) (~407 tok)
- `kvmgr.h` — if !defined(__KVMGR_H__) (~306 tok)
- `misc.h` — if !defined(_MISC_H_) (~88 tok)
- `myconf.h` — if !defined(_MYCONF_H_) (~458 tok)
- `mylock.h` — if !defined(_MYLOCK_H_) (~281 tok)
- `netio.h` — if !defined(_NETIO_H_) (~1293 tok)
- `objs.h` — if !defined(_OBJS_H_) (~36 tok)
- `ordq.h` — if !defined(_ORDQ_H_) (~266 tok)
- `persist.h` — pragma once (~1914 tok)
- `preason.h` — if !defined(_PREASON_H_) (~108 tok)
- `qobj.h` — if !defined(_QOBJ_H_) (~41 tok)
- `repl.h` — pragma once (~979 tok)
- `request.h` — if !defined(_REQUEST_H_) (~2108 tok)
- `resobj.h` — if !defined(_RESOBH_H_) (~46 tok)
- `resreq.h` — if !defined(_RESREQ_H_) (~3223 tok)
- `typemap.h` — if !defined(_TYPEMAP_H_) (~3593 tok)
- `utils.h` — if !defined(_UTILS_H_) (~1574 tok)
- `vtaskmgr.h` — if !defined(_VTASKMGR_H_) (~442 tok)
- `wal_types.h` — pragma once (~679 tok)
- `wobj.h` — if !defined(_WOBBJ_H_) (~42 tok)
- `xcealloc.h` — if !defined(_XCEALLOC_H_) (~146 tok)
- `xceapi.h` — if !defined(_XCEAPI_H_) (~436 tok)
- `xcectx.h` — if !defined(_XCECTX_H_) (~388 tok)
- `xcelimit.h` — if !defined(_XCELIMIT_H_) (~281 tok)
- `xcelog.h` — ifndef _XCELOG_H_ (~819 tok)
- `xlist.h` — if !defined(__XLIST_H__) (~193 tok)
- `xstring.h` — if !defined(__XSTRING_H__) (~54 tok)
- `xvec.h` — if !defined(_XVEC_H_) (~293 tok)
- `zarray.h` — if !defined(_ZARRAY_H_) (~384 tok)
## src/include/xce/
- `cluster_def.h` — ifndef XCE_CLUSTER_DEF_H (~1178 tok)
- `cluster.h` — A single controller candidate. (~1211 tok)
- `errnox.h` — if !defined(__XCE_ERRNOX_H__) (~262 tok)
- `types.h` — < NULL. (~5454 tok)
## src/lib/
- `delay.c` — include <errno.h> (~1798 tok)
- `helper.c` — include <errno.h> (~2956 tok)
- `meson.build` (~1513 tok)
- `netio.c` — include <arpa/inet.h> (~5716 tok)
- `request.c` — include <errno.h> (~12892 tok)
## src/lib/raft/
- `raft_sim.c` — ifdef XCE_TESTING (~1627 tok)
- `raft.c` — include "raft.h" (~2452 tok)
- `raft.h` — ifndef RAFT_H (~3049 tok)
## src/lib/testsuite/
- `test_delay.c` — delay unit tests; adds test_delay_task_ms_fires_within_tolerance and test_delay_task_ms_invalid_args (~1600 tok)
- `test_libxce_retry.c` — define _GNU_SOURCE (~2741 tok)
- `test_netio_on_connected.c` — define _GNU_SOURCE (~1431 tok)
- `test_raft.c` — include <errno.h> (~25610 tok)
- `test.c` — include <argp.h> (~971 tok)
## src/scheduler/testsuite/
- `test.c` — define UNIT_TEST 1 (~4290 tok)
## src/schema/
- `controller.capnp` — Declares OPCode (~8223 tok)
## src/tools/
- `deploy-ha.sh` (~505 tok)
- `test-heartbeat-tolerance.sh` — test-heartbeat-tolerance.sh — Task 4.5: missed-heartbeat tolerance via SIGSTOP/SIGCONT (~2377 tok)
## src/xce.build/
- `build.ninja` — This is the build file for project "xce" (~75089 tok)
## src/xce.build/lib/
- `controller.capnp.c` — include "controller.capnp.h" (~134780 tok)

240
.wolf/buglog.json
View file

@ -5292,6 +5292,246 @@
"related_bugs": [],
"occurrences": 2,
"last_seen": "2026-06-09T08:35:24.717Z"
},
{
"id": "bug-331",
"timestamp": "2026-06-09T09:28:15.795Z",
"error_message": "Significant refactor of ",
"file": "src/controller/myapp.c",
"root_cause": "5 lines replaced/restructured",
"fix": "Rewrote 27→36 lines (5 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-09T09:28:15.795Z"
},
{
"id": "bug-332",
"timestamp": "2026-06-09T09:41:02.753Z",
"error_message": "Significant refactor of ",
"file": "src/lib/raft/raft.c",
"root_cause": "4 lines replaced/restructured",
"fix": "Rewrote 9→15 lines (4 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-09T09:41:02.753Z"
},
{
"id": "bug-333",
"timestamp": "2026-06-09T10:23:01.661Z",
"error_message": "Significant refactor of ",
"file": "src/controller/ha.c",
"root_cause": "37 lines replaced/restructured",
"fix": "Rewrote 41→4 lines (37 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-09T10:23:01.661Z"
},
{
"id": "bug-334",
"timestamp": "2026-06-09T12:56:11.095Z",
"error_message": "Wrong condition in logic",
"file": "src/controller/ha.c",
"root_cause": "Condition was: if (my_role == CTRL_STANDBY && ha_heartbeat_is_fresh(h)",
"fix": "Changed to: if (my_role == CTRL_STANDBY && ha->primary_ctrl_fd >= )",
"tags": [
"auto-detected",
"logic-fix",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-09T12:56:11.095Z"
},
{
"id": "bug-335",
"timestamp": "2026-06-09T00:00:00Z",
"error_message": "UPDATE_RESVALS dequeue writes worker resval at index 0 unconditionally instead of resolving resource idx by name",
"file": "src/scheduler/src/scheduler.c",
"root_cause": "zarray_put(&w->resvals, 0, rv->val) hardcodes index 0; worker resvals are indexed by resource idx (see schedobj.c:291)",
"fix": "NOT YET FIXED — found during scheduler-controller-msg-gap validation gap analysis (research.md G-1). Fix: key_find rv->name in sched->resmgr, put at r->idx",
"tags": [
"scheduler",
"resvals",
"latent-bug",
"gap-analysis",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-09T00:00:00Z"
},
{
"id": "bug-336",
"timestamp": "2026-06-09T00:00:00Z",
"error_message": "XCE_SCHED_OP_UPDATE_RESOURCES cannot update existing resources; makeResource returns NULL/EEXIST and return value is silently ignored",
"file": "src/scheduler/src/scheduler.c",
"root_cause": "makeResource_ in src/lib/resobj.c rejects duplicates with EEXIST; dequeue case ignores NULL return, so 'add or update' (EARS-UB-1.1) is only 'add'",
"fix": "NOT YET FIXED — found during scheduler-controller-msg-gap validation gap analysis (research.md G-2). Fix: on EEXIST mutate existing resource fields and log failures",
"tags": [
"scheduler",
"resources",
"latent-bug",
"gap-analysis",
"silent-error",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-09T00:00:00Z"
},
{
"id": "bug-337",
"timestamp": "2026-06-09T00:02:00Z",
"error_message": "Vote-deny gate inert: ha_heartbeat_is_fresh reads ha->last_heartbeat which has no production writer after the liveness refactor",
"file": "src/controller/ha.c",
"root_cause": "ha-heartbeat-liveness refactor moved activity tracking to ha->last_activity but ha_heartbeat_is_fresh (ha.c:1762) still reads the orphaned last_heartbeat timespec; only tests write it, so freshness is always stale after ~heartbeat_timeout_ms of uptime and standbys grant votes despite a live primary (gates at ha.c:214 and ha.c:1841)",
"fix": "FIXED 2026-06-09: ha_heartbeat_is_fresh now reads ha->last_activity (returns 0 when last_activity==0); dead last_heartbeat field removed from ha_state_t; test_ha_vote_gate.c helper rewritten as set_last_activity_age and fresh/stale tests now also set primary_ctrl_fd. All tests green.",
"tags": [
"ha",
"heartbeat",
"liveness",
"vote-gate",
"dead-field",
"gap-analysis"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-09T17:30:00Z"
},
{
"id": "bug-338",
"timestamp": "2026-06-09T00:00:00Z",
"error_message": "controller_unit_test: test_vote_granted_to_same_candidate_same_term FAIL (Expected 1 Was 0); test_standby_fresh_heartbeat_denies_vote_request FAIL (Expected 0 Was 1, Req 6.2)",
"file": "src/controller/ha.c",
"root_cause": "Uncommitted handle_request_vote changes: (1) duplicate-vote deny made candidate-agnostic (denies same-candidate idempotent re-grant, diverging from raft.c which keeps re-grant); (2) STANDBY vote deny now requires primary_ctrl_fd >= 0, tests use ha_init default fd=-1",
"fix": "FIXED 2026-06-09: handle_request_vote Req 2.6 check restored to same-candidate idempotent re-grant (deny only when voted_for_name differs from candidate_id), matching raft.c; test_standby_fresh_heartbeat_denies_vote_request updated to set primary_ctrl_fd=999. Also fixed myapp.c leader_known check to treat a vote for myconf->name as a self-vote (production records real name, not the 'self' test placeholder). All tests green.",
"tags": [
"ha",
"election",
"vote-gate",
"raft",
"unit-test",
"uncommitted-drift",
"gap-analysis"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-09T17:30:00Z"
},
{
"id": "bug-339",
"timestamp": "2026-06-10T00:23:18.666Z",
"error_message": "Missing guard clause",
"file": "src/controller/ha.c",
"root_cause": "No early return/throw for edge case: last == 0",
"fix": "Added guard clause: if (last == 0)",
"tags": [
"auto-detected",
"guard-clause",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-10T00:23:18.666Z"
},
{
"id": "bug-340",
"timestamp": "2026-06-10T00:23:21.918Z",
"error_message": "Wrong condition in logic",
"file": "src/controller/ha.c",
"root_cause": "Condition was: if (ha->voted_for_term == term && ha->voted_for_name[0)",
"fix": "Changed to: if (ha->voted_for_term == term && ha->voted_for_name[0)",
"tags": [
"auto-detected",
"logic-fix",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-10T00:23:21.918Z"
},
{
"id": "bug-341",
"timestamp": "2026-06-10T00:23:52.563Z",
"error_message": "Wrong reference: last_heartbeat should be voted_for_term",
"file": "src/controller/ha.h",
"root_cause": "Used \"last_heartbeat\" instead of \"voted_for_term\"",
"fix": "Changed last_heartbeat → voted_for_term",
"tags": [
"auto-detected",
"wrong-reference",
"h"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-10T00:23:52.563Z"
},
{
"id": "bug-342",
"timestamp": "2026-06-10T00:24:41.873Z",
"error_message": "Significant refactor of ",
"file": "src/controller/myapp.c",
"root_cause": "4 lines replaced/restructured",
"fix": "Rewrote 13→17 lines (4 removed)",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-10T00:24:41.873Z"
},
{
"id": "bug-339",
"timestamp": "2026-06-09T17:30:00Z",
"error_message": "raft_init discards persisted vote term: voted_for_term reset to 0 even when initial_voted_for is restored, so duplicate-vote check cannot match after restart (possible double vote in same term)",
"file": "src/lib/raft/raft.c",
"root_cause": "raft_init set s->voted_for_term=0 unconditionally; persistence writes (term, voted_for) as a pair, so a restored vote belongs to initial_term",
"fix": "FIXED 2026-06-09: voted_for_term = (initial_voted_for != -1) ? initial_term : 0. Added test_proto_H6_restored_vote_blocks_second_vote and strengthened H1 assertion in src/lib/testsuite/test_raft.c. Found as candidate bug C2 in raft-library gap analysis.",
"tags": [
"raft",
"election-safety",
"restart",
"persistence",
"gap-analysis"
],
"related_bugs": [
"bug-338"
],
"occurrences": 1,
"last_seen": "2026-06-09T17:30:00Z"
},
{
"id": "bug-344",
"timestamp": "2026-06-10T00:32:34.054Z",
"error_message": "Incorrect value in code",
"file": "../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_ha_bootstrap_progress.md",
"root_cause": "Had `ha->last_heartbeat`",
"fix": "Changed to `ha->last_activity`",
"tags": [
"auto-detected",
"wrong-value",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-10T00:32:34.054Z"
}
]
}

9
.wolf/cerebrum.md
View file

@ -29,15 +29,24 @@
- vote_ep_ctx_t lifetime (Task 2.4 review fix): do NOT free vctx in vote_ep_err_handler. ep->data (vctx) is also used by vote_reply_msg_handler. Cleanup deferred to ha_election_cleanup() (task 5.1). Freeing in err_handler causes UAF.
- timeout_ctx field in ha_elect_ctx_t (Task 2.4 review fix): added void *timeout_ctx field pointing to election_timeout_ctx_t. Set after delay_task_ms succeeds. Allows task 4.1 (quorum-reached path) to cancel the watchdog by setting timeout_ctx->cancelled=1 atomically.
- Heartbeat freshness source of truth (2026-06-09): `ha_heartbeat_is_fresh` derives from `ha->last_activity` (whole CLOCK_MONOTONIC seconds, written by ha_on_heartbeat_received, lock-free time_t read). The old `last_heartbeat` timespec field was REMOVED from ha_state_t — do not reintroduce it. The standby vote-deny gate is `role==CTRL_STANDBY && primary_ctrl_fd >= 0 && ha_heartbeat_is_fresh(ha)`; RDHUP resets primary_ctrl_fd to -1 so a standby whose primary died can grant votes immediately even if last_activity is still fresh.
- raft_init restore contract (2026-06-09): persistence writes (term, voted_for) as a pair, so `raft_init` must set `voted_for_term = initial_term` when `initial_voted_for != -1`. Leaving it 0 lets a restarted node double-vote in the same term (Election Safety violation). Covered by test_proto_H6.
## Do-Not-Repeat
<!-- Mistakes made and corrected. Each entry prevents the same mistake recurring. -->
<!-- Format: [YYYY-MM-DD] Description of what went wrong and what to do instead. -->
- [2026-06-09] Do NOT make the duplicate-vote deny candidate-agnostic ("term-only deny"). Raft §5.2 grants iff votedFor ∈ {Nil, candidateId} — same-candidate re-grant must stay idempotent (network retransmission path). An uncommitted ha.c change did this and broke test_vote_granted_to_same_candidate_same_term; ha.c must match raft.c's Step 4 semantics (deny only a DIFFERENT candidate in the same term). See bug-338.
- [2026-06-09] The "self" voted_for_name placeholder only exists in test mode; production raft_cb_persist records the node's REAL name for a self-vote. Any check for "did I vote for myself" must compare against BOTH "self" and c->myconf->name (see myapp.c leader_known).
- [2026-06-04] capnpc-c 1.0.1 has a bug with `:group` inside a `union` when `$C.codecgen` is active: the encode/decode generator emits `d->FIELDNAME` (flat) instead of `d->data.GROUPNAME.FIELDNAME`, AND places the case as dead code after the prior `break`. Do NOT use inline `:group` in a union that has `$C.codecgen`. Instead, define a named outer struct (`struct Foo $C.mapname("foo_t")`) and reference it as a pointer union member. The named struct generates correct `encode_Foo_ptr`/`decode_Foo_ptr` calls. The `read_RequestReply`/`write_RequestReply` wire layer works correctly with groups but the codec layer does not.
## Decision Log
<!-- Significant technical decisions with rationale. Why X was chosen over Y. -->
- Election liveness fix (2026-06-09): `skip_election` path in myapp.c had two bugs: (1) `voted_for_name != ""` triggered even for self-votes and stale terms — fixed by adding `strcmp(name,"self")!=0 && voted_for_term==ha->term`; (2) `ha_arm_heartbeat_timeout_vtask` was never called in skip_election path — fixed by adding arm call after role=CTRL_STANDBY. raft.c Step 4 simplified to term-only deny (same-candidate idempotent re-grant preserved). ha.c `handle_request_vote_req` also simplified to term-only deny.
- CORRECTION (2026-06-09 validation pass): the ha.c "term-only deny" part was WRONG — it denied same-candidate re-grants, diverging from raft.c and failing tests. Reverted to deny-different-candidate-only (see Do-Not-Repeat). The myapp.c "self" check was also incomplete — production self-votes are persisted under the node's real name, so the check now also compares myconf->name.
- Task 4.1 (ha_elect_evaluate_and_signal COMMITTING path): Added cluster_t *c field to ha_elect_ctx_t so evaluate can call cluster_init_phase2 without signature change. election_timeout_ctx_t and committing_timeout_ctx_t must be defined BEFORE ha_elect_evaluate_and_signal. cluster_init_phase2 wrapped in #ifndef XCE_TESTING to avoid linker error (cluster.c not in unit-test binary). ha_phase2_hook seam used in XCE_TESTING builds. On PRIMARY: cancel election timeout watchdog, call phase2 if first_boot, set CTRL_PRIMARY_COMMITTING, reset repl_ack_count=0, arm committing_timeout_fn. On STANDBY: call phase2 if first_boot, set CTRL_STANDBY.

35
.wolf/cron-state.json
View file

@ -1,5 +1,5 @@
{
"last_heartbeat": "2026-06-08T10:03:14.767Z",
"last_heartbeat": "2026-06-10T01:06:27.659Z",
"engine_status": "running",
"execution_log": [
{
@ -167,6 +167,39 @@
"status": "success",
"timestamp": "2026-06-08T09:00:00.899Z",
"duration_ms": 4
},
{
"task_id": "project-suggestions",
"status": "failed",
"timestamp": "2026-06-10T00:19:56.961Z",
"duration_ms": 9810,
"error": "claude -p failed: Exit code 1: SessionEnd hook [export PATH=\"$($SHELL -lc 'echo $PATH' 2>/dev/null):$PATH\"; _R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=$(ls -dt $HOME/.claude/plugins/cache/thedotmack/claude-mem/[0-9]*/ 2>/dev/null | head -1); _R=\"${_R%/}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code session-complete] failed: Hook cancelled"
},
{
"task_id": "token-audit",
"status": "success",
"timestamp": "2026-06-10T00:20:16.187Z",
"duration_ms": 53
},
{
"task_id": "project-suggestions",
"status": "failed",
"timestamp": "2026-06-10T00:20:16.188Z",
"duration_ms": 19227,
"error": "claude -p failed: Exit code 1: SessionEnd hook [export PATH=\"$($SHELL -lc 'echo $PATH' 2>/dev/null):$PATH\"; _R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=$(ls -dt $HOME/.claude/plugins/cache/thedotmack/claude-mem/[0-9]*/ 2>/dev/null | head -1); _R=\"${_R%/}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code session-complete] failed: Hook cancelled"
},
{
"task_id": "cerebrum-reflection",
"status": "failed",
"timestamp": "2026-06-10T00:20:16.188Z",
"duration_ms": 9557,
"error": "claude -p failed: Exit code 1: SessionEnd hook [export PATH=\"$($SHELL -lc 'echo $PATH' 2>/dev/null):$PATH\"; _R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=$(ls -dt $HOME/.claude/plugins/cache/thedotmack/claude-mem/[0-9]*/ 2>/dev/null | head -1); _R=\"${_R%/}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code session-complete] failed: Hook cancelled"
},
{
"task_id": "anatomy-rescan",
"status": "success",
"timestamp": "2026-06-10T01:00:00.580Z",
"duration_ms": 50
}
],
"dead_letter_queue": [],

20
.wolf/daemon.log
View file

@ -91,3 +91,23 @@
[2026-06-08T08:03:14.768Z] [INFO] Dashboard server listening on port 18791
[2026-06-08T09:00:00.895Z] [INFO] Executing task: Consolidate old memory
[2026-06-08T09:00:00.900Z] [INFO] Task Consolidate old memory completed in 4ms
[2026-06-10T00:06:27.655Z] [INFO] Scheduled task: Full anatomy rescan (0 */6 * * *)
[2026-06-10T00:06:27.657Z] [INFO] Scheduled task: Consolidate old memory (0 2 * * *)
[2026-06-10T00:06:27.657Z] [INFO] Scheduled task: Token audit report (0 0 * * 1)
[2026-06-10T00:06:27.657Z] [INFO] Scheduled task: Cerebrum reflection (0 3 * * 0)
[2026-06-10T00:06:27.657Z] [INFO] Scheduled task: AI suggestions (0 4 * * 1)
[2026-06-10T00:06:27.659Z] [INFO] File watcher started on .wolf/
[2026-06-10T00:06:27.659Z] [INFO] OpenWolf daemon started
[2026-06-10T00:06:27.660Z] [INFO] Dashboard server listening on port 18791
[2026-06-10T00:07:13.306Z] [INFO] WebSocket client connected
[2026-06-10T00:12:13.316Z] [INFO] WebSocket client connected
[2026-06-10T00:19:47.150Z] [INFO] Executing task: AI suggestions
[2026-06-10T00:19:56.961Z] [ERROR] Task AI suggestions failed (attempt 1): claude -p failed: Exit code 1: SessionEnd hook [export PATH="$($SHELL -lc 'echo $PATH' 2>/dev/null):$PATH"; _R="${CLAUDE_PLUGIN_ROOT}"; [ -z "$_R" ] && _R=$(ls -dt $HOME/.claude/plugins/cache/thedotmack/claude-mem/[0-9]*/ 2>/dev/null | head -1); _R="${_R%/}"; [ -z "$_R" ] && _R="$HOME/.claude/plugins/marketplaces/thedotmack/plugin"; node "$_R/scripts/bun-runner.js" "$_R/scripts/worker-service.cjs" hook claude-code session-complete] failed: Hook cancelled
[2026-06-10T00:19:56.961Z] [INFO] Executing task: AI suggestions
[2026-06-10T00:20:06.631Z] [INFO] Executing task: Cerebrum reflection
[2026-06-10T00:20:16.134Z] [INFO] Executing task: Token audit report
[2026-06-10T00:20:16.188Z] [INFO] Task Token audit report completed in 53ms
[2026-06-10T00:20:16.188Z] [ERROR] Task AI suggestions failed (attempt 1): claude -p failed: Exit code 1: SessionEnd hook [export PATH="$($SHELL -lc 'echo $PATH' 2>/dev/null):$PATH"; _R="${CLAUDE_PLUGIN_ROOT}"; [ -z "$_R" ] && _R=$(ls -dt $HOME/.claude/plugins/cache/thedotmack/claude-mem/[0-9]*/ 2>/dev/null | head -1); _R="${_R%/}"; [ -z "$_R" ] && _R="$HOME/.claude/plugins/marketplaces/thedotmack/plugin"; node "$_R/scripts/bun-runner.js" "$_R/scripts/worker-service.cjs" hook claude-code session-complete] failed: Hook cancelled
[2026-06-10T00:20:16.188Z] [ERROR] Task Cerebrum reflection failed (attempt 1): claude -p failed: Exit code 1: SessionEnd hook [export PATH="$($SHELL -lc 'echo $PATH' 2>/dev/null):$PATH"; _R="${CLAUDE_PLUGIN_ROOT}"; [ -z "$_R" ] && _R=$(ls -dt $HOME/.claude/plugins/cache/thedotmack/claude-mem/[0-9]*/ 2>/dev/null | head -1); _R="${_R%/}"; [ -z "$_R" ] && _R="$HOME/.claude/plugins/marketplaces/thedotmack/plugin"; node "$_R/scripts/bun-runner.js" "$_R/scripts/worker-service.cjs" hook claude-code session-complete] failed: Hook cancelled
[2026-06-10T01:00:00.530Z] [INFO] Executing task: Full anatomy rescan
[2026-06-10T01:00:00.580Z] [INFO] Task Full anatomy rescan completed in 50ms

140
.wolf/hooks/_session.json
View file

@ -1,80 +1,110 @@
{
"session_id": "session-2026-06-09-2117",
"started": "2026-06-09T04:17:13.321Z",
"session_id": "session-2026-06-10-1720",
"started": "2026-06-10T00:20:07.286Z",
"files_read": {
"/home/rshen/projects/xce/worktrees/persistent_ha/src/schema/controller.capnp": {
"count": 1,
"tokens": 8223,
"first_read": "2026-06-09T06:41:14.575Z"
"/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c": {
"count": 4,
"tokens": 25610,
"first_read": "2026-06-10T00:20:59.235Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/cluster.c": {
"count": 2,
"tokens": 12219,
"first_read": "2026-06-09T06:43:58.698Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/include/clusterconfig.h": {
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.h": {
"count": 1,
"tokens": 474,
"first_read": "2026-06-09T06:46:31.070Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/config_repl.c": {
"count": 1,
"tokens": 14728,
"first_read": "2026-06-09T06:47:59.696Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/myconf.c": {
"count": 1,
"tokens": 1100,
"first_read": "2026-06-09T07:31:53.498Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test.c": {
"count": 1,
"tokens": 18744,
"first_read": "2026-06-09T07:31:53.527Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/startup_validation.c": {
"count": 1,
"tokens": 1423,
"first_read": "2026-06-09T08:04:51.033Z"
},
"/home/rshen/projects/xce/worktrees/persistent_ha/src/docs/cluster-config-split-analysis.md": {
"count": 1,
"tokens": 2796,
"first_read": "2026-06-09T08:32:42.474Z"
"tokens": 10007,
"first_read": "2026-06-10T00:23:31.412Z"
}
},
"files_written": [
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/docs/cluster-config-split-analysis.md",
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c",
"action": "edit",
"tokens": 308,
"at": "2026-06-09T08:33:08.519Z"
"tokens": 114,
"at": "2026-06-10T00:22:18.260Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/docs/cluster-config-split-analysis.md",
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c",
"action": "edit",
"tokens": 577,
"at": "2026-06-09T08:33:34.327Z"
"tokens": 523,
"at": "2026-06-10T00:22:44.317Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/docs/cluster-config-split-analysis.md",
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/testsuite/test_raft.c",
"action": "edit",
"tokens": 618,
"at": "2026-06-09T08:35:24.714Z"
"tokens": 36,
"at": "2026-06-10T00:22:57.391Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/docs/cluster-config-split-analysis.md",
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/lib/raft/raft.c",
"action": "edit",
"tokens": 165,
"at": "2026-06-09T08:36:14.677Z"
"tokens": 144,
"at": "2026-06-10T00:23:00.680Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.c",
"action": "edit",
"tokens": 260,
"at": "2026-06-10T00:23:18.664Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.c",
"action": "edit",
"tokens": 121,
"at": "2026-06-10T00:23:21.916Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.h",
"action": "edit",
"tokens": 85,
"at": "2026-06-10T00:23:49.279Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/ha.h",
"action": "edit",
"tokens": 28,
"at": "2026-06-10T00:23:52.561Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_vote_gate.c",
"action": "edit",
"tokens": 122,
"at": "2026-06-10T00:24:14.357Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_vote_gate.c",
"action": "edit",
"tokens": 109,
"at": "2026-06-10T00:24:18.132Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/testsuite/test_ha_vote_gate.c",
"action": "edit",
"tokens": 63,
"at": "2026-06-10T00:24:21.425Z"
},
{
"file": "/home/rshen/projects/xce/worktrees/persistent_ha/src/controller/myapp.c",
"action": "edit",
"tokens": 258,
"at": "2026-06-10T00:24:41.870Z"
},
{
"file": "/home/claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_ha_bootstrap_progress.md",
"action": "edit",
"tokens": 147,
"at": "2026-06-10T00:32:34.052Z"
}
],
"edit_counts": {
"src/docs/cluster-config-split-analysis.md": 4
"src/lib/testsuite/test_raft.c": 3,
"src/lib/raft/raft.c": 1,
"src/controller/ha.c": 2,
"src/controller/ha.h": 2,
"src/controller/testsuite/test_ha_vote_gate.c": 3,
"src/controller/myapp.c": 1,
"../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_ha_bootstrap_progress.md": 1
},
"anatomy_hits": 8,
"anatomy_hits": 2,
"anatomy_misses": 0,
"repeated_reads_warned": 1,
"repeated_reads_warned": 3,
"cerebrum_warnings": 0,
"stop_count": 3
"stop_count": 1
}

87
.wolf/memory.md
View file

@ -2,7 +2,9 @@
> Chronological action log. Hooks and AI append to this file automatically.
> Old sessions are consolidated by the daemon weekly.
| 09:45 | Fixed election liveness: (1) raft.c step-4 deny is now term-only (same-candidate idempotent re-grant preserved); (2) ha.c vote gate also term-only; (3) myapp.c skip-election guard adds voted_for_name!="self" + voted_for_term==term; (4) watchdog armed after skip_election | raft.c, ha.c, myapp.c | FIXED | ~200 |
| 09:12 | Task 6.6 residual fix: removed double-free in ha_disarm_heartbeat_timeout_vtask (line 3424 free(ctx) removed); deferred free now handled solely by liveness_poll_fn; build+tests GREEN | ha.c | FIXED | ~80 |
| 09:45 | Fix test_proto_B5_idempotent_same_candidate: modified raft_on_vote_request Step 4 to allow idempotent re-grant when same candidate re-requests vote in same term; different candidate still denied; all tests GREEN | lib/raft/raft.c | FIXED | ~60 |
| 21:20 | Updated cluster-config-split-analysis.md: added decisions to drop port fallback and remove candidates from ClusterConfig; detailed per-file impact table; revised implementation phases (Phase 1 = immediate removal, Phases 2-5 = split/migration) | src/docs/cluster-config-split-analysis.md | DONE | ~120 |
| 07:46 | Task 6.6 (ha-repl-hardening): integration test for snapshot transfer on lagging standby; found and fixed 2 ASAN bugs: double-free in ha_connect_to_peer (bug-006) and heap-UAF in liveness_poll_fn on ctx re-arm (bug-007); confirmed snapshot trigger path (oldest_wal_lsn=1 > applied_lsn=0 → requesting snapshot transfer, checkpoint_lsn=0, size=16) fires correctly after standby kill+restart; all 3 nodes stable, no ASAN errors; standby-side apply blocked by known capnp payload deferral limitation | ha.c | GREEN (trigger confirmed, apply deferred by design) | ~800 |
| 06:30 | Task 6.4 (ha-repl-hardening): audited snapshot chunk transfer tests; tests 2+3 already covered (reconnect discards snap.part, lsn<oldest_wal triggers snapshot); added test_snapshot_recv_128kb_two_chunks (131072-byte receive path, 2×65536 chunks, verifies total_size/rename/applied_lsn=checkpoint_lsn); registered in dispatch; 1/1 controller test OK ASAN | test_config_repl.c | GREEN | ~400 |
@ -695,3 +697,88 @@
| 01:35 | Edited src/docs/cluster-config-split-analysis.md | added 1 condition(s) | ~577 |
| 01:36 | Edited src/docs/cluster-config-split-analysis.md | 6→7 lines | ~154 |
| 01:36 | Session end: 4 writes across 1 files (cluster-config-split-analysis.md) | 8 reads | ~61375 tok |
## Session: 2026-06-09 01:59
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-09 02:09
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 02:27 | Edited src/lib/raft/raft.c | modified if() | ~129 |
| 02:27 | Edited src/controller/ha.c | strcmp() → deny() | ~60 |
| 02:28 | Edited src/controller/myapp.c | modified if() | ~497 |
| 02:28 | Edited src/controller/myapp.c | modified if() | ~281 |
| 02:41 | Edited src/lib/raft/raft.c | voted() → grant() | ~210 |
| 02:46 | Session end: 5 writes across 3 files (raft.c, ha.c, myapp.c) | 5 reads | ~72268 tok |
| 03:17 | Edited src/controller/ha.c | added 3 condition(s) | ~497 |
| 03:22 | Edited src/controller/ha.h | expanded (+15 lines) | ~199 |
| 03:22 | Edited src/controller/ha.c | added 2 condition(s) | ~530 |
| 03:23 | Edited src/controller/ha.c | removed 41 lines | ~43 |
| 03:23 | Edited src/controller/ha.c | removed 33 lines | ~2 |
| 03:23 | Edited src/controller/myapp.c | 5→10 lines | ~121 |
| 03:26 | Session end: 11 writes across 4 files (raft.c, ha.c, myapp.c, ha.h) | 8 reads | ~84261 tok |
| 04:02 | Session end: 11 writes across 4 files (raft.c, ha.c, myapp.c, ha.h) | 9 reads | ~84766 tok |
| 05:56 | Edited src/controller/ha.c | 2→3 lines | ~47 |
| 06:00 | Session end: 12 writes across 4 files (raft.c, ha.c, myapp.c, ha.h) | 10 reads | ~84423 tok |
| 06:02 | Session end: 12 writes across 4 files (raft.c, ha.c, myapp.c, ha.h) | 10 reads | ~84423 tok |
| 06:06 | Edited src/lib/raft/raft.c | modified if() | ~178 |
| 06:10 | Session end: 13 writes across 4 files (raft.c, ha.c, myapp.c, ha.h) | 11 reads | ~84614 tok |
## Session: 2026-06-09 16:49
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-09 16:51
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-09 16:52
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 14:xx | Validation-pass gap analysis appended for worker-join-refactor | .kiro/specs/worker-join-refactor/research.md | implemented, only 5.4 test-run unverified | ~9k |
| 16:59 | Edited .kiro/specs/resval-id-decoupling/research.md | modified and() | ~2232 |
| 00:02 | Validation-pass gap analysis appended for scheduler-controller-msg-gap (found G-1 resvals idx-0 bug, G-2 resource EEXIST, leaks, test debt) | .kiro/specs/scheduler-controller-msg-gap/research.md | partial, not 100% | ~12k |
| 00:02 | Gap analysis validation pass for ha-heartbeat-liveness; found dead last_heartbeat field feeding vote gate | .kiro/specs/ha-heartbeat-liveness/research.md, .wolf/buglog.json | appended analysis + bug entry | ~25k |
| 00:03 | Gap analysis validation pass for ha-config-bootstrap spec (50 ACs verified against on-disk code) | .kiro/specs/ha-config-bootstrap/research.md | appended analysis: ~40/50 implemented, 7 partial, 3 pivoted | ~45k |
| 16:55 | Gap analysis validation pass for ha-peer-connection-reuse (18/19 reqs Implemented, 5.5 Partial: ASAN leaks); ran controller_unit_test (769 PASS/0 FAIL) | .kiro/specs/ha-peer-connection-reuse/research.md | appended analysis | ~30k |
| 00:13 | gap-analysis validation pass: election-iomgr-refactor — found 2 failing unit tests from uncommitted ha.c vote changes | .kiro/specs/election-iomgr-refactor/research.md | appended analysis + buglog entry | ~30k |
| 00:13 | Ran /kiro-validate-gap across all 9 specs via 9 parallel agents; appended Gap Analysis Validation Pass to each research.md | .kiro/specs/*/research.md | found 1 critical (repl data plane), 4 bugs logged (335-338) | ~1130k |
| 17:14 | Created ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_ha_bootstrap_progress.md | — | ~576 |
| 17:14 | Edited ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/MEMORY.md | inline fix | ~54 |
| 17:15 | Session end: 3 writes across 3 files (research.md, project_ha_bootstrap_progress.md, MEMORY.md) | 50 reads | ~194368 tok |
## Session: 2026-06-10 17:19
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-10 17:19
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-10 17:20
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 17:22 | Edited src/lib/testsuite/test_raft.c | 7→10 lines | ~106 |
| 17:22 | Edited src/lib/testsuite/test_raft.c | modified test_proto_H6_restored_vote_blocks_second_vote() | ~488 |
| 17:22 | Edited src/lib/testsuite/test_raft.c | 1→2 lines | ~34 |
| 17:23 | Edited src/lib/raft/raft.c | 4→8 lines | ~135 |
| 17:23 | Edited src/controller/ha.c | added 1 condition(s) | ~243 |
| 17:23 | Edited src/controller/ha.c | 5→8 lines | ~113 |
| 17:23 | Edited src/controller/ha.h | 4→4 lines | ~79 |
| 17:23 | Edited src/controller/ha.h | inline fix | ~26 |
| 17:24 | Edited src/controller/testsuite/test_ha_vote_gate.c | set_last_heartbeat_age() → set_last_activity_age() | ~114 |
| 17:24 | Edited src/controller/testsuite/test_ha_vote_gate.c | 2→7 lines | ~102 |
| 17:24 | Edited src/controller/testsuite/test_ha_vote_gate.c | set_last_heartbeat_age() → set_last_activity_age() | ~59 |
| 17:24 | Edited src/controller/myapp.c | 13→17 lines | ~241 |
| 17:45 | Fixed bug-337 (freshness→last_activity), bug-338 (same-candidate re-grant + myapp self check), bug-339 (raft_init voted_for_term restore); full build+tests green; reopened ha-repl-hardening tasks 7.1-7.7; marked worker-join-refactor + resval-id-decoupling implemented | ha.c ha.h myapp.c raft.c test_raft.c test_ha_vote_gate.c tasks.md spec.json | all tests pass, 0 FAIL | ~80k |
| 17:32 | Edited ../../../../../claude/.claude/projects/-home-rshen-projects-xce-xce/memory/project_ha_bootstrap_progress.md | 3→3 lines | ~138 |
| 17:32 | Session end: 13 writes across 7 files (test_raft.c, raft.c, ha.c, ha.h, test_ha_vote_gate.c) | 2 reads | ~37627 tok |

2238
.wolf/token-ledger.json
View file

File diff suppressed because it is too large Load diff