Recovered .wolf/buglog.json (828 entries lost when this worktree's copy
silently reset to bug-001 sometime after commit ce96fd6d, 3 days
uncommitted with zero overlap detection) by merging the committed
history back in and renumbering the 43 real entries logged since.
To stop this recurring:
- OPENWOLF.md: require searching for an existing same-file/root_cause
entry before appending a new bug id (fixes the bug-1418 x4 duplication
pattern found in the recovered history); never restart the id counter.
- stop.js: validate buglog.json every turn (JSON validity, {version,bugs}
wrapper, duplicate ids, count/max-id not regressing vs. the last
commit) and auto-commit it standalone when it grew cleanly, so git
history is always a recoverable backup instead of a 3-day-stale one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
controller_unit_test crashed deterministically in write_WalCatchupEntry
(test_wal_catchup_append_term_fields_round_trip) -- the same bug class
as bug-1504 (SnapshotData.internalState): this codebase's capnpc-c
backend silently no-ops Data-field encoding in generated encode_XXX()
composite helpers without zero-initializing the caller's local struct
first, leaving an uninitialized capn_ptr written onto the wire.
Confirmed zero production exposure before fixing: xce_wal_batch_encode_
raw()/wal_entry_to_wire() (request.c) never call the generic
encode_WalCatchupEntry() at all -- they memset() their own struct and
call write_WalCatchupEntry() directly with every field hand-set, so
this was 100% test-only exposure (grepped every caller of the generic
encode/decode pair; only the crashing test uses it).
Fix mirrors bug-1504: WalCatchupEntry.payload (@2) and batchPayloads
(@10) switched from Data to List(UInt8) in controller.capnp (new
batchPayloadsLen @11 companion; payload reuses the existing
wirePayloadLen @9), typemap.h's wal_catchup_entry_t field types
updated, and request.c's extract_data_bytes() widened to take a raw
capn_ptr (it already only used .p internally) so its call sites work
regardless of which Data-shaped wrapper the generated accessor
returns now. batchPayloads' single-top-level-blob shape confirmed
this does not reintroduce bug-847's per-element cross-segment
far-pointer loss -- that issue is inherent to per-element data
regardless of Data vs List(UInt8) spelling.
Strengthened test_wal_catchup_append_term_fields_round_trip with real
payload/batchPayloads content assertions -- previously never
exercised, since the test's src fields were left NULL.
Verified: controller suite 3/3 (was 2/3, crashing), scheduler suite
unaffected at 37/37 (unrelated schema area).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the final task of "Core: Interactive REPL" (Req 5 AC 22): proves
the REPL's own documented invariant -- "step/continue to completion
produces byte-identical output to batch --sim" (design.md SimDriver
Invariants).
test_repl_equivalence.c runs the same sched_sim_driver_run() on the
same 3-session/9-decision fixture twice: once batch (interactive=
false), once REPL-driven via freopen()'d scripted "step 1\ncontinue\n"
stdin into the real repl_command_loop()/readline() path (same
mechanism test_repl_execution_control.c established) -- then
structurally decodes both runs' --sim-output and compares decision
content. Test-only; zero production code changed.
RED-phase probe (planted in xce_repl_on_event(), reverted, zero
residue) confirmed the comparison is real: corrupting one job's id
under interactive mode broke exactly this test plus the 3 other REPL
tests that share the same hook, none else.
Scheduler suite: 37/37 clean (was 36/36). Task group 12 (Interactive
REPL, tasks 12.1-12.4) is now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds `snapshot [FILE]` and `set speed|output|verify ...`/`info config`
(Req 5 AC 22 "Snapshots"/"Simulation parameters") to the interactive
simulation REPL.
`snapshot` reuses sim_record.c's existing sim_record_write_header_and_
snapshot() (task 11.5) rather than reimplementing capnp encoding, so
the produced file is genuinely loadable via a fresh --sim invocation
-- the task's own Observable. `set speed`/`set output`/`set verify`
mutate fields sim_loop.c and core_clock.c already read fresh on every
event/session, so no new hook points were needed and sim_loop.c stays
untouched at exactly 800/800 lines (this project's hard per-file cap).
New repl_params.c is a fourth sibling in the repl.c/repl_break.c/
repl_info.c split (task 12.2's precedent) -- neither existing file had
headroom left under the 800-line cap for this task's own logic.
Scheduler suite: 36/36 clean (was 35/35). Review independently
reproduced the RED-phase probe and additionally verified the
snapshot round-trip Observable is non-vacuous.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the interactive simulation REPL's breakpoint and state-inspection
commands (Req 5 AC 22): `break <event_type> [key=value ...]` / `info
break` / `delete <id>|all` on event and decision types, and `info
session/job/queue/worker/resource/summary`.
repl.c (370 lines after task 12.1) couldn't hold this task's full
surface under the project's 800-line cap, so it's split three ways
(matching the snapshot.c/snapshot_wire.c/snapshot_load.c precedent):
repl.c (execution control + dispatch), repl_break.c (matching/CRUD +
the three new driver-facing hooks), repl_info.c (info dumps), sharing
state via repl_internal.h. sim_loop.c gained 3 new bare hook calls
(repl.h inline wrappers, same pattern as task 12.1's two) at the
points design.md's pseudocode requires -- per decoded event before it
joins the batch, per session boundary, and per drained decision --
landing sim_loop.c at exactly 800/800 lines.
Two real bugs found and fixed during development (bug-1506, bug-1507):
an uninitialized xstr_table left breakpoint filter fields silently
NULL instead of failing loud, and repl_breakpoint_free() called a bare
free() on an RTTI-refcounted makeString() string, corrupting the heap.
Review found one gap: `info job` printed only a resreq presence flag
instead of the actual resreq text, when format_resreq() already
existed in this repo for exactly this purpose. Fixed to match.
Scheduler suite: 35/35 clean (was 34/34). RED-phase evidence (2 real
bugs plus 2 synthetic probes, all reverted with confirmed zero
residue) recorded in tasks.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the interactive simulation REPL's execution-control slice (Req 5
AC 22): a single-threaded input loop (readline-based) supporting
step [N], continue, quit, and help, pausing before the first replay
event and blocking the driver loop while active.
Review round 1 raised two findings, both resolved here:
- GNU readline (GPL) was linked directly into xce_scheduler, a shipped
production daemon. Switched the whole project to libedit (BSD) for
line editing -- scheduler and controller (xce_test) both now use
editline/readline.h, API-compatible (readline()/add_history()).
- Getting a clean regression run after the libedit swap required a
full clean rebuild, which exposed a genuine pre-existing SIGSEGV
(bug-1504): SnapshotData.internalState was a capnp Data field, and
this codebase's capnpc-c backend silently no-ops Data-field
encoding without zero-initializing the caller's stack struct,
leaving an uninitialized capn_ptr written onto the wire. Fixed by
switching internalState to List(UInt8) (with a new internalStateLen
wire field), matching the working FetchFileReply.content/bytesRead
precedent -- no hand-written accessor code needed.
A second, unrelated instance of the same bug class was found in the
controller's WAL-catchup path (WalCatchupEntry.payload) and
deliberately left unfixed as bug-1505, out of scheduler-core's
boundary; needs its own project-wide Data-field audit.
Scheduler suite: 34/34 clean. RED-phase reproduction recorded in
tasks.md (repl_should_pause() step-budget check disabled -> 2/4
tests fail as expected -> reverted -> 34/34 green).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds xce_sched_core_save_snapshot()/xce_sched_core_load_snapshot(),
split across snapshot.c/snapshot_wire.c/snapshot_load.c (800-line cap)
along real save/load/shared-helper boundaries. Serializes the full
core state (config, jobs, queues, workers, resources, resvals, users,
projects, prng_seed, internal_state) into/out of sched_snapshot_data_t
(task 7.3's committed C mirror of the capnp SnapshotData struct) as
plain in-memory structures -- no wire encoding here, that's a later
task. After load, rebuilds the picker's private state by re-invoking
its state-change hooks (queue updates, then job pushes) against the
restored core state, matching xce_core_push_events()'s own hook-firing
order; picker-private state itself is never stored in the snapshot.
Adds a new prng_seed field to xce_sched_core_t -- requirements.md/
design.md both presuppose it but no prior task added it. Seeded to 0,
no PRNG consumer wired yet (deliberately out of scope).
Cross-core behavioral-equivalence proven directly: two independent
cores, each with its own registered picker, save on one and load into
a freshly-initialized other, both driven through an identical real
session, decision sequences asserted identical in order.
Found and fixed two real bugs along the way: a missing prng_seed
restore in this task's own new code, and a genuine guard bug in
already-committed core.c (sched_get_user()/sched_get_project()
treating key_find()'s expected -ENOENT as fatal, silently breaking
get-or-create for any new name) that also affects the live production
path (make_sched_job(), JOBS_PUSH) -- a narrowly-scoped two-line fix,
independently traced to the primitive level and confirmed correct by
review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
buglog.json: corrected bug-1480/1481 (stale "not yet fixed" entries, code
landed same day the entries were written); bug-1483/1484 fixed; bug-1485
investigated, downgraded to low-priority/environment-flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
30-round chaotic soak reported committed_job_loss=1 at round 25, but the
flagged job was confirmed present on every later verify and the final
sweep -- a real loss never comes back. Root-caused from ctrl-1/ctrl-2/ctrl-3
logs, no controller change needed: a newly-elected primary cannot fold a
prior-term entry into its own committed_lsn merely because it holds it
durably (Raft Figure-8 safety via ha_arm_own_term_gate) -- it needs a fresh
majority ack for its own-term WAL_NOOP first. That normally closes in
<200ms, but chaotic mode's overlapping-restart design let it stretch to
6.232s this run (the peer whose ack was needed was itself mid-catch-up from
its own recent restart). verify_classify ran its expect-job check
immediately on primary detection with no settle/retry, unlike verify_no_loss
(bug-1422) or wait_for_full_convergence, so it landed inside that window.
Added a bounded retry (5 attempts, 2s settle) around the LOSS classification
only, mirroring verify_no_loss's precedent -- a job still missing after that
still counts as a genuine loss. Verified with a fresh 30-round chaotic
re-soak: committed_divergence=0 committed_job_loss=0 result=PASS.
Verified 190 tests across 8 files cover Requirements 2.4, 3.1-3.4,
4.1, 4.4, 7.1-7.3, 7.5, 8.1-8.4. 1067 tests, 0 failures -- unchanged,
zero-diff verification task.
Task 7.2's sweep surfaced a real, pre-existing gap (bug-1421, not
this feature's fault): no test anywhere drives the real
repl_ack_decode_and_dispatch path and asserts term adoption fires for
the heartbeat-reply or WAL-NACK term-discovery vectors -- only the
config-push vector this feature added (task 2.2) is actually tested.
Logged as bug-1484; outside this spec's task list to fix.
Adds test_5_1_wal_catchup_reject_sends_nack_reply, driving a genuine
(term,lsn) conflict against a real persisted WAL entry through
ha_apply_term_entry_via_library and asserting
ha_send_wal_ack_reply_ex(success=0) fires via ha_wal_ack_send_hook.
Note: test_config_repl.c's test_bug1479_ae_reject_sends_nack_not_silence
already covers the same call site via a different (forward-gap)
stimulus, added in the original bug-1479 commit. design.md's own File
Structure Plan explicitly calls for this new test in this file
regardless, so this is disclosed, design-sanctioned defense-in-depth,
not accidental duplication -- both tests independently confirmed to
fail together when the reply call is removed.
1065 -> 1066 tests, 0 failures.
Found during raft-adapter-conformance task 3.2; not fixed (out of that
task's boundary), avoided via the existing stub-and-reset workaround
already used by test_ha_prevote.c's LOST-path tests.
