fix(controller): stop SIGSEGV in WalCatchupEntry capnp encode (bug-1505)

controller_unit_test crashed deterministically in write_WalCatchupEntry
(test_wal_catchup_append_term_fields_round_trip) -- the same bug class
as bug-1504 (SnapshotData.internalState): this codebase's capnpc-c
backend silently no-ops Data-field encoding in generated encode_XXX()
composite helpers without zero-initializing the caller's local struct
first, leaving an uninitialized capn_ptr written onto the wire.

Confirmed zero production exposure before fixing: xce_wal_batch_encode_
raw()/wal_entry_to_wire() (request.c) never call the generic
encode_WalCatchupEntry() at all -- they memset() their own struct and
call write_WalCatchupEntry() directly with every field hand-set, so
this was 100% test-only exposure (grepped every caller of the generic
encode/decode pair; only the crashing test uses it).

Fix mirrors bug-1504: WalCatchupEntry.payload (@2) and batchPayloads
(@10) switched from Data to List(UInt8) in controller.capnp (new
batchPayloadsLen @11 companion; payload reuses the existing
wirePayloadLen @9), typemap.h's wal_catchup_entry_t field types
updated, and request.c's extract_data_bytes() widened to take a raw
capn_ptr (it already only used .p internally) so its call sites work
regardless of which Data-shaped wrapper the generated accessor
returns now. batchPayloads' single-top-level-blob shape confirmed
this does not reintroduce bug-847's per-element cross-segment
far-pointer loss -- that issue is inherent to per-element data
regardless of Data vs List(UInt8) spelling.

Strengthened test_wal_catchup_append_term_fields_round_trip with real
payload/batchPayloads content assertions -- previously never
exercised, since the test's src fields were left NULL.

Verified: controller suite 3/3 (was 2/3, crashing), scheduler suite
unaffected at 37/37 (unrelated schema area).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ

.wolf/buglog.json

@@ -14821,7 +14821,7 @@
     "error_message": "controller_unit_test SIGSEGV in write_WalCatchupEntry (lib/controller.capnp.c), test_wal_catchup_append_term_fields_round_trip -- discovered incidentally while chasing bug-1504; NOT fixed, logged for its own follow-up.",
     "file": "src/schema/controller.capnp",
     "root_cause": "Same root cause as bug-1504 (Data-field codegen no-op leaves an uninitialized stack capn_ptr in the generated encode_XXX_ptr()'s local struct), a DIFFERENT field: WalCatchupEntry.payload (@2: Data), in the controller's HA/WAL-catchup subsystem (bug-847's own territory -- see controller.capnp's comment there), entirely unrelated to scheduler-core. Confirms this is a systemic, project-wide bug class affecting every `:Data` field in controller.capnp whose bytes are populated via the generic encode_XXX()/write_XXX() composite pipeline rather than fully hand-written low-level accessor code -- not a one-off. Known/suspected other Data fields with the same latent gap: ReplayHeader.configHash (dormant so far, no real consumer), CtrlHelloRequest.key, and possibly others not yet audited.",
-    "fix": "NOT FIXED. Deliberately deferred -- out of scheduler-core task 12.1's boundary (this is the controller's HA/WAL-replication subsystem), and a proper fix requires a project-wide audit of every `:Data` field in controller.capnp to decide case-by-case: convert to List(UInt8) (bug-1504's fix, when the field is genuinely a flat byte blob with no cross-segment/per-element concerns) vs. keep Data with real hand-written low-level accessor code at every encode/decode call site (matching the batchPayloads precedent, when List(UInt8) risks bug-847's per-element far-pointer loss for repeated/batch structures). Needs its own scoped task, not a one-off patch.",
+    "fix": "FIXED 2026-08-07. Investigated production exposure first: xce_wal_batch_encode_raw()/wal_entry_to_wire() (request.c) never call the generic encode_WalCatchupEntry() at all -- they memset() their own struct and call write_WalCatchupEntry() directly with every field hand-set, deliberately leaving payload unset (bug-847's own comment). So this bug was 100% test-only exposure, zero production risk, confirmed by grepping every caller of encode_WalCatchupEntry_ptr/decode_WalCatchupEntry (only test_e2e_capnp.c uses the generic path). Fix: same as bug-1504 -- changed WalCatchupEntry.payload (@2) and batchPayloads (@10) from Data to List(UInt8) in controller.capnp, with batchPayloadsLen (@11, new) as batchPayloads' maplistcount companion (payload reuses the existing wirePayloadLen @9). Verified batchPayloads' own top-level-single-blob shape means this does not reintroduce bug-847's per-element cross-segment far-pointer loss (that issue is inherent to PER-ELEMENT data regardless of Data vs List(UInt8) spelling -- payload, the per-element field, was already legacy/unpopulated in production before this fix and stays that way; batchPayloads is single top-level, never subject to that issue either way). Updated typemap.h's wal_catchup_entry_t field types (xce_data_t* -> uint8_t*, new batch_payloads_wire_len). Fixed request.c's extract_data_bytes() to accept a raw capn_ptr instead of capn_data (it already only ever used .p internally) so all 4 call sites (WalCatchupEntry.payload/batchPayloads, SnapshotTransfer.payload, per-sub-entry we.payload) work regardless of which Data-shaped wrapper type the accessor returns. Strengthened test_wal_catchup_append_term_fields_round_trip (test_e2e_capnp.c) with real payload/batchPayloads content assertions (previously never exercised at all, since the test's src.payload/batch_payloads were left NULL). Verified: controller_unit_test no longer crashes (3/3 controller suite), scheduler suite unaffected (37/37, unrelated schema area).",
     "tags": [
       "controller",
       "ha",
@@ -14839,7 +14839,7 @@
       "bug-847"
     ],
     "occurrences": 1,
-    "last_seen": "2026-08-07T08:10:00Z"
+    "last_seen": "2026-08-07T09:15:00Z"
   },
   {
     "id": "bug-1506",

.wolf/cerebrum.md

@@ -722,3 +722,9 @@ Root-caused a `committed_job_loss=1` chaotic-soak FAIL entirely from logs/timest
 ## Key Learning (2026-08-07, scheduler-core task 12.4) — a RED-phase probe for a "REPL never alters WHAT is processed, only WHEN" invariant must corrupt CONTENT, not TIMING: defeating repl_should_pause() (task 12.1's own already-verified probe, which flips the `step N` pause decision) does NOT turn a REPL-vs-batch decision-sequence equivalence test red, because it only changes when stdin is read/control returns — every session still eventually runs to completion regardless of pause schedule, and any script built on CONTINUE's sticky non-pausing behavior tolerates arbitrary pause timing. The correct probe has to touch a hook that can affect content and fires ONLY under `sim->interactive`: `xce_repl_on_event(sim, ev)` (repl_break.c) takes a non-const `xce_core_event_t*` and is called strictly before the event joins the batch `xce_core_push_events()` later applies — a one-line mutation there (e.g. corrupting a JOBS_PUSH job's jobid) is invisible to batch mode entirely and produces a genuine, real divergence in the recorded output file. Confirmed the probe correctly also failed 3 OTHER pre-existing REPL tests that exercise JOBS_PUSH under interactive mode (test_repl_execution_control.c/test_repl_snapshot_params.c/test_repl_breakpoints_inspection.c) — a real production bug at this hook site would legitimately be caught by the whole REPL test family, not just one file, which is expected and reassuring, not a signal of over-broad blast radius from the probe itself.
 
 ## Decision Log (2026-08-07, scheduler-core task 12.4) — task 12.4's "byte-identical decision sequences" is a DIFFERENT comparison than task 11.6's test_sim_equivalence.c: 11.6 compares the NORMAL-mode driver against the SIMULATION driver on the same recorded fixture; 12.4 compares the SAME simulation driver (sched_sim_driver_run()) against itself, run twice on the identical `.replay` fixture, differing only in `sim->interactive` (REPL-driven via scripted stdin vs straight batch). Both runs set `sched_sim_args_t.output_path` directly (confirmed unconditional on `interactive` — sim.c's fopen() gate has no interactive check), so no `set output` REPL command is needed; multi-session fixtures (more than one ReplaySession boundary, needed to meaningfully exercise BOTH `step` and `continue` per the task's literal text) require scanning the whole output file structurally for ReplaySession-shaped messages (sim.c's shared msg_root_has_shape(), each `.replay` message type has a distinct (word,ptr) struct shape) rather than reusing test_sim_equivalence.c's fixed-skip-count decode technique (valid only for exactly one session).
+
+## Key Learning (2026-08-07, bug-1505 fix) — before fixing a capnp Data-field no-op crash (see the earlier Key Learning on this bug class, scheduler-core task 12.1), always check whether ANY production caller reaches the generic `encode_XXX_ptr()` composite path at all, before assuming the fix is production-critical. `xce_wal_batch_encode_raw()`/`wal_entry_to_wire()` (`src/lib/request.c`) `memset()` their own wire struct then hand-write every field and call `write_WalCatchupEntry()` DIRECTLY (bypassing `encode_WalCatchupEntry()`/`encode_WalCatchupEntry_ptr()` entirely) — this codebase's own established pattern for any Data-shaped field it actually needs to populate correctly (also the pattern `xce_wal_batch_encode_raw()`'s own comment cites: "Bypasses encode_WalCatchupEntry ... Builds the Request manually"). `memset()`-first is what makes this safe: an unset field degrades to a legitimate null `capn_ptr`, not garbage. Grep every caller of the generic `encode_X_ptr`/`decode_X` pair before fixing — if only test code calls it, the fix is test-hygiene + defense-in-depth, not an emergency; still worth fixing (a future caller could hit it), but changes the urgency/framing when reporting to the user.
+
+## Key Learning (2026-08-07, bug-1505 fix) — when converting a capnp field from `Data` to `List(UInt8)` (the bug-1504/1505 fix pattern) fixes a field ALSO passed through hand-written low-level accessor code elsewhere (e.g. `extract_data_bytes(capn_data dp, ...)` in `request.c`, called on multiple Data-shaped fields including ones NOT being converted), the accessor's parameter type must widen too. `capn_data`/`capn_list8` are both trivial single-field `{ capn_ptr p; }` wrappers with zero behavioral difference — the cleanest fix is changing the accessor to take a raw `capn_ptr` and updating every call site to pass `.p` explicitly, rather than picking one wrapper type and forcing the other call sites to construct a throwaway wrapper struct just to satisfy the compiler.
+
+## Do-Not-Repeat (2026-08-07, bug-1505 fix) — before assuming a `List(UInt8)` conversion for a PER-ELEMENT field (inside a `List(Struct)`) is safe from bug-847's cross-segment far-pointer loss, re-read that bug's own root cause carefully: it's about per-element Data/List(UInt8) fields losing their far-pointer when a multi-entry batch spans capnp segments — a property of being PER-ELEMENT, completely independent of whether the schema spells the field `Data` or `List(UInt8)` (both are the identical wire type in capnp; the keyword only changes whether THIS project's capnpc-c backend auto-generates real composite encode/decode for it or silently no-ops it). Converting a per-element field's spelling does not reintroduce or worsen bug-847 — but also does not fix it. Only a genuinely TOP-LEVEL single blob (like `batchPayloads`) is inherently immune to that specific cross-segment issue, which is why bug-847's own original fix routed traffic through a top-level field instead of trying to "fix" the per-element one.