Commit graph

90 commits

Author SHA1 Message Date
4dcfa80ae8
chore(wolf): update session tracking state
anatomy.md/cerebrum.md/memory.md/token-ledger.json/hooks state from the
day's session covering bug-1584/1585/1587-1596 and task 12.5.
2026-08-19 09:49:04 +00:00
a4b5fbbc28
save wolf status 2026-08-18 14:50:44 +00:00
90be04bd92
commit wolf status 2026-08-17 06:35:54 +00:00
c6f20ba553
chore(wolf): checkpoint session state (anatomy, cerebrum, memory, scan-state)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-16 11:02:17 +00:00
ce96fd6d93 save wolf status 2026-08-10 18:01:29 +08:00
2329dbf2a8
feat(scheduler-core): implement REPL snapshot capture and simulation parameter commands (task 12.3)
Adds `snapshot [FILE]` and `set speed|output|verify ...`/`info config`
(Req 5 AC 22 "Snapshots"/"Simulation parameters") to the interactive
simulation REPL.

`snapshot` reuses sim_record.c's existing sim_record_write_header_and_
snapshot() (task 11.5) rather than reimplementing capnp encoding, so
the produced file is genuinely loadable via a fresh --sim invocation
-- the task's own Observable. `set speed`/`set output`/`set verify`
mutate fields sim_loop.c and core_clock.c already read fresh on every
event/session, so no new hook points were needed and sim_loop.c stays
untouched at exactly 800/800 lines (this project's hard per-file cap).

New repl_params.c is a fourth sibling in the repl.c/repl_break.c/
repl_info.c split (task 12.2's precedent) -- neither existing file had
headroom left under the 800-line cap for this task's own logic.

Scheduler suite: 36/36 clean (was 35/35). Review independently
reproduced the RED-phase probe and additionally verified the
snapshot round-trip Observable is non-vacuous.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
2026-08-07 10:25:32 +00:00
4153b35b22 chore(wolf): update buglog and session state
buglog.json: corrected bug-1480/1481 (stale "not yet fixed" entries, code
landed same day the entries were written); bug-1483/1484 fixed; bug-1485
investigated, downgraded to low-priority/environment-flake.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
2026-08-02 04:37:36 +00:00
b17d272f1d add wolf state 2026-07-31 17:42:34 +08:00
d968bbd722 chore(wolf): update session memory, anatomy, and token ledger 2026-07-31 09:29:48 +00:00
34ab78d72d fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:

1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
   library's nextIndex backoff and immediately repl_start_catchup's the
   standby at the corrected position, instead of the 16-strike silent
   eviction with no catch-up driver after it. Correct from the start, but
   live soak still failed identically: nothing ever produced the NACK this
   code was waiting for.

2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
   path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
   silently dropping the reject. ha_send_wal_ack_reply had hardcoded
   success=1 unconditionally -- the only WalAckReply sender in the
   controller -- making (1)'s NACK-routing correct but unreachable dead
   code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
   the NACK without a live endpoint. Live-confirmed the NACK now reaches
   the primary -- but the scenario still failed for a third reason.

3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
   catch-up range was hard-bounded at committed_lsn, which can never advance
   in this exact scenario because advancing it requires an ACK from the very
   standby that's stuck catching up (2-of-3 alive, this standby is the only
   path to a new majority) -- a self-referential deadlock. Widens the
   STREAM_WAL ceiling (and the convergence check gating it) to the leader's
   own log tail, matching raft.tla's real AppendEntries semantics
   (leaderCommit is advisory, never a send-cap), while reusing bug-1478's
   exact guard so this doesn't reopen that trap (never widen onto a trailing
   local-only checkpoint's own unstreamable LSN).

Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-29 04:11:32 +00:00
102e263cce fix(ha): close permanent catch-up wedge from duplicated replication-target formula (bug-1478)
Two independent call sites duplicated target = max(committed_lsn,
last_written): ha.c's heartbeat/stateReport dispatch (repl_ack_decode_and_
dispatch) and config_repl.c's catchup_derive_target (used by the ACK-driven
catch-up continuation path). Once config-upgrade-snapshot's marker-triggered
checkpoint advances last_written one LSN past an already-committed
committed_lsn (the checkpoint's own consumed, never-replicated LSN slot),
the unconditional max() permanently wedges every standby's catch-up target
at an unreachable value -- the primary re-streams an empty range every
heartbeat forever.

Extracted the shared, pure ha_compute_repl_target_lsn(committed_lsn,
last_written) (ha.h/ha.c), which only falls back to last_written on genuine
committed_lsn==0 bootstrap, and pointed both call sites at it so they can
never disagree again.

Fixing the second call site exposed three pre-existing tests in
test_convergence_verification.c that hardcoded the old buggy formula's
output as their expected target; updated their fixture values (committed_lsn
4->9, matching appended_lsn) to decouple them from the now-fixed edge case,
which already has dedicated pure-function coverage in test_ha_role.c.

GREEN: full controller_unit_test suite = 1040 tests, 0 failures, clean
across multiple runs. Live-verified via run-ha-soak.sh
--config-upgrade-wal-trim (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3
using the preserved floor segment) -- unblocks config-upgrade-snapshot
task 8.9.
2026-07-28 09:53:20 +00:00
cae4ef0cc7 save wolf state and document 2026-07-23 09:20:59 +00:00
b87d5e68d3 commit wolf status 2026-07-23 12:30:53 +08:00
60c25c39c4 chore(openwolf): update session memory, buglog, and anatomy tracking
Records bug-1461/1462 fixes, admin-config-tool completion, and
config-upgrade-snapshot spec work across anatomy.md, buglog.json,
cerebrum.md, memory.md, and the session/token-ledger state files.
2026-07-23 04:29:12 +00:00
4ad31d774d save status 2026-07-21 08:24:44 +00:00
619f41f70e chore: update OpenWolf session state (bug-1452 fix)
buglog.json: bug-1452 marked FIXED with root cause and fix writeup.
cerebrum.md: Do-Not-Repeat entry for the commit_engine_fini/persist_stop
ordering contract, plus the flag-noted latent duplicate in
test_demotion_convergence.c.
memory.md/anatomy.md/session state: routine session bookkeeping.
2026-07-19 09:27:56 +00:00
e1f4d676de chore: update OpenWolf session state (xce-audit-invariants controller pass) 2026-07-19 08:14:19 +00:00
aeb6e37ce7 update wolf and project status 2026-07-18 14:22:02 +00:00
8f614b6421 chore: update OpenWolf session state (Group 1 validate-impl batch + bug-1447/1449) 2026-07-18 13:28:10 +00:00
781b0f7ef0 chore: update OpenWolf session state + cerebrum learning (Unity test-harness fix)
Records the controller_unit_test nested UNITY_BEGIN/UNITY_END fix
(commit a81f2a2) in cerebrum.md for future sessions, plus routine
anatomy/memory/token-ledger session bookkeeping.
2026-07-17 14:49:25 +00:00
e6956ecf46 chore: update OpenWolf session state 2026-07-17 14:16:50 +00:00
6592f9cbf1 chore: update OpenWolf session state + cerebrum learnings (meson parallel-timeout false-positive, /tmp/xce_* cleanup) 2026-07-17 13:07:00 +00:00
3f54ef307c chore: update OpenWolf session state (ha-term-safety + controller-status validation) 2026-07-17 11:01:58 +00:00
1a0960bfcf chore: update OpenWolf session state (bug-1445 fix) 2026-07-17 10:42:23 +00:00
fb701c24a1 chore: update OpenWolf session state (job-array-single-wal-record validation pass) 2026-07-17 10:06:10 +00:00
ccee18b0c9 chore: update OpenWolf session state 2026-07-17 09:28:59 +00:00
9176164f4d chore: update OpenWolf session state (anatomy, cerebrum, memory, token ledger) 2026-07-17 07:31:11 +00:00
b3219635b5 chore(wolf): session trail + xce-test-ha cross-run collision guard
run-ha-soak.sh: scope every pgrep/pkill to this run's --workdir, not just
binary name + --myname. An unscoped pkill -9 -x xce_controller (old
cleanup()) killed every xce_controller on the machine, including a
different concurrently-running soak's live primary — root cause of the
first bug-1436 misdiagnosis (a killed-out-from-under-it primary produced
a genuine, correctly-reported ECONNRESET that looked like a libxce bug).

.wolf/*: anatomy/cerebrum/buglog updated for bug-1435/1436, session and
token-ledger trail.
2026-07-13 07:19:00 +00:00
976a6c956b chore(wolf): session state trail 2026-07-12 04:25:53 +00:00
9c020f2680 chore(wolf): session state trail for bug-1433/1434 audit pass
Anatomy/memory/token-ledger updates from the /xce-audit-invariants controller
run and its rerun (bug-1433/1434 fix + verification).
2026-07-11 12:16:33 +00:00
ea0ac81c92 save wolf state and document 2026-07-09 12:00:27 +00:00
c68935127c chore(wolf): restore buglog.json, log bug-1421/bug-1422, memory trail
buglog.json had been reset to a small auto-detected-noise-only file at some
point this session, losing the real 738-entry history (still present in git
HEAD). Restored the original list verbatim and appended bug-1421 (reusing
the ID a prior session had already reserved for this exact election
regression before the file got wiped) and bug-1422, both logged with full
root-cause/fix detail in this branch's other commits.

memory.md carries the OpenWolf session action log (auto-appended by hooks
plus this session's manual entries) for both fixes.
2026-07-09 11:57:19 +00:00
0eb105431b fix(raft-apply-decoupling): bug-1419 layer 2 — checkpoint_lsn mislabeled from WAL tip instead of applied cursor
persist_checkpoint() stamped checkpoint_lsn from p->current_lsn (physical
WAL append tip) instead of p->last_applied_lsn (the true "jobmgr reflects
up through here" cursor), uncoordinated with the async apply pump
introduced by raft-apply-decoupling. write_snapshot() serializes live
jobmgr state, so a checkpoint taken while apply lags the WAL tip mislabels
the snapshot: on restart last_applied_lsn is re-seeded from checkpoint_lsn
and replay resumes strictly above it, silently skipping committed-but-not-
yet-applied entries jobmgr never received. This is the actual mechanism
behind kill-mid-apply producing a permanent APPLY_DIVERGENT fence —
apply_committed_entry() is deterministic/idempotent and write_snapshot()
is crash-atomic, so the bug was a mislabeled watermark, not a structural
divergence requiring a safety/liveness tradeoff.

Fix stamps checkpoint_lsn from last_applied_lsn and holds apply_pump_lock
across the label capture + write_snapshot() so the label and the jobmgr
state it describes are captured atomically w.r.t. the live apply pump.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
2026-07-08 00:05:19 +00:00
a5e29a6ffa fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2)
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.

Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.

Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
2026-07-07 03:03:40 +00:00
17b210c382 chore: clean buglog.json junk/duplicates, gitignore scratch soak output
buglog.json had two broken auto-detection hook signatures polluting it:
- 207 entries "Missing error handling in X" / fix "Added try/catch block"
  (nonsensical on .md/.capnp files, and C has no try/catch — never a real
  applied fix)
- 452 entries with a literally-empty template ("Significant refactor of "
  with no subject ever filled in) / fix "Rewrote N->M lines"
Removed both (659 total). Also merged 24 legitimate duplicate bug IDs
(re-logged/updated entries sharing an ID) into single entries — kept the
most recent content as primary, unioned tags/related_bugs, summed
occurrences — and stripped noise tags ('auto-detected', 'refactor') that
had leaked onto real entries from a since-merged junk duplicate. Net:
1406 -> 721 entries, verified zero remaining duplicate IDs and zero
remaining junk signatures.

.gitignore: added patterns for the scratch soak-run logs and codegraph.html
this session generated at repo root (already deleted from the tree), and
fixed the ucai-formatter-cache.local.json pattern to match its nested
occurrences (.claude/skills/*/.claude/, src/*/.claude/) instead of only the
repo-root one.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
2026-07-03 03:24:19 +00:00
cf3fa9b850 chore: sync OpenWolf state (buglog, cerebrum, memory, anatomy)
Records today's session: bug-1386/1389 snapshot pipeline fixes, bug-795
closure, bug-1400/1401 fixes, bug-1396 harness instrumentation, and the
raft-apply-decoupling spec-family promotion.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
2026-07-03 02:41:41 +00:00
15be5d94e8 fix(controller): extract job-submit commit barrier out of the hook fan-out — stop committed-job-loss (bug-1306)
raft-apply-decoupling item #1 (down payment). xce_run_hooks returns only the
LAST-registered hook's rc, so a later JOB AFTER_EVENT subscriber (sched/
apiserver) returning 0 clobbered persist_job_hook_fun's commit -ETIMEDOUT.
handle_jobsubmit then replied 'jobsubmit ok' for an entry that never reached
majority, and a new-term leader legally discarded it (committed-job-loss,
Commit Durability violation).

- persist.c/.h: new persist_submit_wal_and_commit(p, jlist) — the explicit raft
  step (WAL append + majority-commit barrier) returning the commit verdict
  directly. persist_job_hook_fun's WAL_MODE_WAIT branch is now a no-op.
- job.c handle_jobsubmit: call the explicit step, read commit_rc DIRECTLY,
  reject on <0 before firing the now notification-only AFTER hook
  (canonical commit -> notify -> reply).
- tests: +test_job_hook_new_is_notify_only regression pin; 4 hook tests
  relocated to drive the explicit step. persist_hooks 19/0, e2e_hooks 3/0,
  commit_barrier_reply 22/0, commit_durability 38/0.

Proven: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code).
Residual soak FAILs (divergence catch-up lag + a round-4 election-escalation
tail) are NOT job loss — tracked as task 5.10.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
2026-07-01 00:42:30 +00:00
3609f468c5 chore(raft-ha-core-safety): init spec scaffold (HA-core failover-safety gate)
New spec for the pre-existing HA-core consistency/convergence failures the
xce-test-ha soak reproduces under chaotic failover — committed-prefix
divergence (bug-851, bug-795/828), standby-rejoin/reconverge failure
(bug-1042), and round-N election stall — all baseline-confirmed at commit
21e26a1 (independent of raft-drive-async). This spec is the gate that must
close before raft-drive-async flips its dispatch flags ON in production.

Scaffold only (spec.json + requirements.md project description); requirements
generation is the next phase.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
2026-06-27 03:16:17 +00:00
294b9ce8eb chore(raft-drive-async): task 4.2 regression sweep — no new regression (baseline-confirmed)
Verdict: raft-drive-async introduces NO new regression.

- Unit: controller_unit_test EXIT=0, xcelib_test 282/0 (all HA suites green;
  dispatch/snapshot/configpush flags default OFF, inert).
- Live kill-based failover soak on a real fs (~/tmp), release + ASAN builds:
  zero committed-JOB loss every completed round (jobs 3->12 monotonic).
- The 6-round CHAOTIC soak does not run clean to round 6 (fails ~round 4), but a
  baseline comparison against pre-Phase-3 binaries (commit 21e26a1) fails round 4
  IDENTICALLY (election stall / reconverge). The round-4 committed-prefix
  divergence at LSN 9 is the documented pre-existing bug-851/795 family
  (chaotic-failover silent divergence), present before Phase 3 and out of scope
  for this spec. Logged as bug-1225 (baseline-confirmed).

Also: make the xce-test-ha soak driver's controller/xce_test paths
env-overridable (XCE_HA_BIN / XCE_HA_XT, backward-compatible defaults) — used to
run the baseline binaries through the same driver for the regression comparison.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
2026-06-27 03:04:23 +00:00
d2830ce514 test(raft-drive-async): receiver-side dispatch liveness integration tests (task 3.4b)
With the dispatch flags flipped ON in-test, prove the three receiver-side
liveness properties (production defaults stay OFF):

- P1: while a snapshot (and config-push) transfer is parked in flight on the
  slow lane, another endpoint's peer-control message is still processed
  promptly on the vtask pool (different thread), with the transfer still
  parked — the offloaded transfer does not block the drive path.
- P2: a peer-control ack (classified VTASK_LIVENESS) runs promptly even while
  the pool is saturated to the LivenessAdmission reservation boundary with
  REQ_BLOCK tasks (one more REQ_BLOCK fast-rejects with -E_XCE_NOT_READY,
  proving real saturation). Narrowed to the liveness-dispatch-under-saturation
  assertion; commit-advance itself is covered by the commit-engine tests.
- P3: while one endpoint's peer-control body is parked (holding a worker + that
  ep busy), another endpoint's body runs concurrently on a different worker —
  per-endpoint FIFO does not serialize across endpoints.

All deterministic (cond-var gates, bounded polls). Fixed a flaky teardown-race
hang (bug-1224): the tests released a parked vtask body then immediately called
the cancel-based vtaskmgr_fini while the just-released body was still unwinding
(P2's REQ_BLOCK bodies racing blk_lock teardown the worst). Fix adds end-of-body
completion counters polled to the full expected count before fini, so every
pool body has fully returned (worker idle) before the cancel. slowlane_fini is
graceful (join) so the lane side was already safe.

Verified: 5/5 deterministic isolated runs; full controller suite EXIT=0;
ASan-clean. Completes task 3.4 (3.4a UAF fix + 3.4b tests); the production
flag-flip remains deferred to after the 4.1 partition soak + 4.2 regression
sweep.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
2026-06-27 00:36:22 +00:00
6d510d8cf4 fix(raft-drive-async): endpoint-quiescence for bug-1197 vote-ep teardown (task 3.4a)
When the (still-default-OFF) peer-control dispatch flag is ON,
raft_cb_become_follower runs as a vtask pool worker and frees every peer
vote-ep while other pool workers may still hold those eps' data — a UAF
(bug-1197). This makes that path quiescence-safe so the flag can later be
flipped ON (deferred to the 4.x prod-flip gate; production stays OFF here).

Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO +
busy serialization):

- Deferred teardown-as-vtask: the loser loop, when dispatch is ON and the pool
  is wired, does per vote-ep endpoint_mark_tearing_down -> iomgr_unmonitor ->
  enqueue a VTASK_IO/VTASK_LIVENESS teardown vtask bound to that ep. Per-ep busy
  FIFO makes it run strictly after any in-flight/queued same-ep peer-control
  vtask; the body NULLs iod->ep before endpoint_fini+free(ep) to disarm
  run_task's post-body endpoint_unblock_tasks (which would touch the freed
  ep->lock). OFF / pool-unwired path is byte-for-byte the legacy endpoint_fini.

- RCU-like drain closes the dispatch/teardown TOCTOU window: new _Atomic int
  endpoint_t.dispatch_inflight (+ tearing_down:1 bit). The ON dispatch brackets
  its flag-check+enqueue with fetch_add/fetch_sub (seq_cst);
  endpoint_mark_tearing_down sets the flag under ep->lock, releases, then spins
  until dispatch_inflight drains to 0. So every dispatch that read the flag
  false finishes its enqueue before mark returns (FIFO-before the teardown
  vtask); any later dispatch reads the flag set and drops the packet. ep->lock
  is deliberately NOT held across the enqueue (it inverts with the taskq lock in
  vtask_eligible); the counter is the lock-free handshake. Drain is bounded: the
  held window is just a calloc + non-blocking enqueue on the single io_worker.

Still open before the production flag flip (4.x): the borrowed-ep race in
snapshot_lane_fn/configpush_lane_fn is distinct (slowlane thread vs io_worker
RDHUP teardown, not the vtask-pool FIFO) and needs its own per-standby-ep
discipline.

Tests: test_ha_vep_teardown.c — 5 ASan tests (ordering, late-event drop, run_task
unblock disarm, OFF-path unchanged, deterministic mark-drains-inflight-dispatch).
xcelib_test 282/0; controller_unit_test EXIT=0; ASan-clean. Adversarial
cpp-review: no CRITICAL/HIGH; handshake formally sound, flag-ON path race-free.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
2026-06-26 17:37:26 +00:00
bd02064ebc feat(raft-drive-async): offload requester-less config-push to slow lane (task 3.3)
push_config_transport_only spawns one thread per standby then joins them with a
CONF_PUSH_TIMEOUT_MS bounded wait — blocking the caller. On the requester-less
drive-thread paths (promotion + heartbeat convergence) that wait stalls the
io_worker. Route those through ha_config_push_offloadable, gated by
g_drive_async_configpush_offload (default OFF → inline, byte-for-byte legacy;
ON → slow lane, non-blocking).

Flag ON deep-copies the config bytes + sha into a lane ctx (caller keeps owning
and freeing its own buffer); a single cluster-wide single-flight guard
(ha_state_t.configpush_inflight scalar CAS + lane key=1) bounds it to one
config-push fan-out in flight; the in-flight flag is cleared on every terminal
path (two alloc failures, -EEXIST, hard submit error, lane completion). Send
body factored into ha_config_push_do_send.

Three requester-less drive-thread call sites converted: raft_cb_become_leader
(promotion), the HA_REPL_PUSH_CONFIG heartbeat case, and the direct-CTRL_PRIMARY
FSM promotion branch. The admin-upgrade barrier (push_config_to_standbys, has a
waiting requester) and join.c are untouched.

Lands DORMANT like tasks 3.1/3.2: the lane reads c->ha.standby_endpoints and
writes those borrowed standby endpoints, racing io_worker RDHUP teardown if run
off-thread (bug-1197 family). Flag OFF keeps push+teardown serialized on the
io_worker. Task 3.4 flips this ON together with the snapshot and peerctrl flags
after establishing endpoint-lifetime discipline.

Tests (controller suite EXIT=0): test_configpush_offload_runs_on_lane,
test_configpush_inline_when_flag_off, test_configpush_offload_dedups.
Adversarial cpp-review: no CRITICAL/HIGH/MEDIUM findings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
2026-06-26 16:05:54 +00:00
6c0523b69b feat(raft-drive-async): snapshot SlowLane offload + WAL-trim cursor guard (task 3.2)
Part B (LIVE, default-ON): trim_wal_segments now bounds WAL-segment deletion
by repl_trim_lsn(&p->repl, active_seg_lsn) = min(active segment LSN, every
active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn),
after repl_expire_disconnect_slots ages out stale slot guards. floor <=
active_seg_lsn structurally, so the guard can only preserve more segments,
never delete one a live standby still needs — closes the bug-795/828
trim-vs-cursor race for connected/recently-disconnected standbys (bug-1147
stopgap; empty/cursor=0 join still needs entity-bearing snapshots).

Part A (DORMANT, g_drive_async_snapshot_offload default OFF): ha_snapshot_send_
dispatch routes HA_REPL_SEND_SNAPSHOT inline (flag OFF — byte-for-byte legacy,
on io_worker) or to slowlane_submit+snapshot_lane_fn (flag ON). Send body
factored into ha_snapshot_do_send. Per-peer single-flight via
ha_state_t.snapshot_inflight[] CAS + slowlane key; in-flight flag cleared on
all four terminal paths, mirroring the reconnect offload (task 2.1).

Lands dormant like task 3.1: snapshot_lane_fn borrows a standby endpoint it
does not own, so flag-ON would race io_worker RDHUP teardown (cross-thread UAF,
bug-1197 family). Flag OFF keeps send+teardown serialized on the io_worker.
Task 3.4 is the single gate that flips both this and the peerctrl flag ON after
establishing endpoint-lifetime discipline; bug-1198 stays open by design until
then. The offload flag was initially authored default-ON (would ship the UAF)
and corrected to OFF in ha.c and the ha.h accessor doc (bug-1210).

Tests (controller 196/0, 0 new leaks): test_snapshot_offload_runs_on_lane,
test_snapshot_inline_when_flag_off, test_snapshot_offload_dedups_per_peer,
test_trim_guard_preserves_standby_segments,
test_trim_without_standbys_proceeds_normally. Adversarial cpp-review: GO.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
2026-06-26 15:43:24 +00:00
295769d8ae add wolf state 2026-06-25 14:30:46 +08:00
af099dca3e test(ha-convergence-recovery): close Task 8 — live /xce-test-ha 8-round PASS
Ran the live 3-node E2E (the last open task). Result:
  PASS: 8 rounds, 30 jobs submitted, zero job loss; all 3 nodes led
  (ctrl-1,ctrl-2,ctrl-3)  (exit 0)

Live-covered: kill-primary -> promoted standby owns all confirmed jobs
(zero loss across 8 failovers, jobs 3->30, expect-job ok each round);
restart ex-primary -> converged rejoin, 3-of-3 each round, no phantom jobs;
deterministic leadership coverage (ctrl-3 forced via RequestVote-freshness
fence). The soak also served as the live regression for bug-1077 (~40
failover client round-trips, no SEGV) after rebuilding the controller (the
deployed binaries predated that fix).

Not exercised live (remain unit-covered, tasks 2/4/5): withhold-ACK barrier
single-outcome (test_demotion_convergence) and isolate/heal quorum-WARN
(test_convergence_verification). All 8 tasks now [x]; spec closed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
2026-06-24 10:17:11 +00:00
8da8fd03dd fix(libxce): bug-1077 ctrlstatus crash from shared reply-union type confusion
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).

Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
  never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
  and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
  one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.

Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
2026-06-24 09:44:43 +00:00
61e1391cd0 fix bug-1060 bug-1067 2026-06-24 17:05:43 +08:00
0eafed3999 save wolf status 2026-06-23 07:08:43 +00:00
20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00
223e1b8578 docs(libxce-failover): record live-E2E round-2 fixes (bug-609 EINVAL, bug-610 reconnect SEGV) 2026-06-17 06:54:28 +00:00