myxce/.kiro/specs/ha-entity-snapshots/tasks.md

# Implementation Plan

- [x] 1. Foundation: entity-snapshot test harness
- [x] 1.1 Build a reusable test fixture for a persist_t backed by a populated job/queue table
  - Model on the existing `make_hook_test_persist` / `cbrp_setup` fixture patterns in the test
    suite
  - Fixture can seed N jobs (varying state) and M queues (varying status) into the cluster's
    job/queue tables
  - Observable: a new test file exists and compiles, exposing a helper that produces a ready
    persist_t+cluster_t with seeded entities
  - _Requirements: 1.1, 2.1_

- [x] 2. Core: entity-bearing checkpoint write path — jobs
- [x] 2.1 RED: job checkpoint content, consistency, and failure-discard tests
  - Assert a checkpoint containing N jobs round-trips through load with all N job records
    field-identical
  - Assert a job actively transitioning state on a concurrent thread during the checkpoint walk
    is captured as a single consistent pre- or post-transition value, never torn
  - Assert a simulated write/encode failure partway through leaves no file at the final
    checkpoint path — the same discard behavior that (per design) is what already makes the
    existing trim/rotate sequencing satisfy the retention-safety requirement
  - Observable: new tests compile and fail (RED) against the current sentinel-only write_snapshot
  - _Requirements: 1.1, 1.2, 1.3, 3.1, 3.2, 3.3_
  - _Boundary: Persistence Layer (write_snapshot)_
- [x] 2.2 Collect and encode live jobs into the checkpoint under per-entity consistency
  - Enumerate live jobs under the existing retention policy, take each job's own read lock
    nested inside the table walk, encode via the existing job serializer, and append to the
    checkpoint after the current header
  - Any failure at any step discards the temp file and leaves no file at the final path
  - Observable: the RED tests from 2.1 pass
  - _Requirements: 1.1, 1.2, 1.3, 1.4_
  - _Boundary: Persistence Layer (write_snapshot)_

- [x] 3. Core: entity-bearing checkpoint write path — queues
- [x] 3.1 RED: queue admin-status round-trip test
  - Assert a queue's (name, status) pair written into the checkpoint and read back matches
    exactly
  - Observable: new test compiles and fails (RED)
  - _Requirements: 1.1_
  - _Boundary: Persistence Layer (write_snapshot)_
  - _Depends: 2.2_
- [x] 3.2 Collect queue admin-status into the checkpoint
  - Enumerate every queue present in the live table (its table presence already implies
    config-provisioned existence), take each queue's own read lock nested inside the table walk,
    append its name+status as a plain record (no serialization schema needed — matches the
    existing restore function's exact contract)
  - Observable: the RED test from 3.1 passes; a checkpoint containing M queues restores M
    correct statuses
  - _Requirements: 1.1, 1.2_
  - _Boundary: Persistence Layer (write_snapshot)_

- [x] 4. Core: protect an in-flight checkpoint transfer from rotation
- [x] 4.1 (P) Track which checkpoint position each in-flight peer transfer references
  - Add a per-peer tracking array alongside the existing in-flight-transfer flag; set it when a
    transfer begins, clear it when the transfer completes
  - Observable: a unit test confirms the tracked position is set before a transfer's first byte
    goes out and cleared after completion
  - _Requirements: 3.1, 3.2, 3.3_
  - _Boundary: HA Replication Layer_
- [x] 4.2 RED: rotation-guard test
  - Assert a checkpoint file whose position matches an in-flight transfer survives file rotation
    even when it would otherwise be rotated out
  - Observable: new test compiles and fails (RED) against the current rotation logic
  - _Requirements: 3.1, 3.2, 3.3_
  - _Depends: 4.1_
- [x] 4.3 Skip rotating out a checkpoint file that is currently mid-transfer
  - Extend the existing file-rotation pass to check the in-flight tracking before deleting a
    candidate file
  - Observable: the RED test from 4.2 passes
  - _Requirements: 3.1, 3.2, 3.3_

- [x] 5. Core: entity restoration on load
- [x] 5.1 RED: restore-before-advance, failure-safety, and legacy-format tests
  - Assert loading a checkpoint with N jobs + M queues restores all of them before the member's
    position advances
  - Assert a deliberately corrupted entity record causes the load to fail with the member's
    position left unchanged
  - Assert a checkpoint file predating this feature (position marker only) is accepted without
    crashing, reports itself as position-only, and does not falsely claim full restoration
  - Observable: new tests compile and fail (RED) against the current watermark-only load
  - _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2_
  - _Boundary: Persistence Layer (load_snapshot)_
  - _Depends: 2.2, 3.2_
- [x] 5.2 Restore entities before advancing the member's position, with legacy-format detection
  - Detect whether a checkpoint carries entity content or is a legacy position-only file
  - Restore every job then every queue status via the existing restore functions, in file order;
    on any failure, stop and leave position unchanged
  - Only after every entity restores successfully, advance the member's committed position;
    extend the load function's contract with an indicator of whether entities were actually
    restored
  - Observable: the RED tests from 5.1 pass
  - _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2_
  - _Boundary: Persistence Layer (load_snapshot)_

- [x] 6. Integration: wire the catch-up receive path to the new restore contract
- [x] 6.1 Consume the new restoration indicator in the snapshot-chunk receive handler
  - The handler advances the member's applied position only after the load succeeds (unchanged
    ordering); on a legacy (position-only) file it logs the fallback rather than treating it as
    a failure
  - Observable: given a legacy-format file, the member logs the fallback and continues through
    normal catch-up; given an entity-bearing file, the member's applied position advances only
    after restoration succeeds
  - _Requirements: 2.4, 4.1, 4.2, 5.1, 5.2_
  - _Depends: 5.2_

- [x] 7. Validation: chaotic catch-up soak proves end-to-end correctness
- [x] 7.1 Extend the chaotic HA failover soak to force at least one full-checkpoint catch-up round
  - Configure or trigger a round where a member rejoins empty or far enough behind that it must
    receive a full checkpoint rather than incremental history
  - Observable: the soak verdict for that round reports zero committed-job loss and zero
    committed-prefix divergence, and existing comparison tooling confirms the rejoined member's
    job/queue state matches the rest of the cluster
  - _Requirements: 4.1, 4.2, 4.3_
  - _RESOLVED 2026-07-01 (raft-ha-core-safety task 5.10a, same investigation — see that spec's
    task 5.10 note for the full chain). The blocking `walcompare` per-member checkpoint-floor gap
    was fixed (commit `f3c86a6`). The "specific soak run never actually exercised a full-checkpoint
    catch-up at all" gap was closed by building a `--force-snapshot` soak mode (commit `9f4292d`)
    plus a `--wal-segment-bytes` CLI override (this session) — the WAL segment physical size was a
    hardcoded 64 MB compile-time constant, and `trim_wal_segments()` never deletes the currently-
    open segment, so at soak job counts a segment never rotated and nothing was ever trimmable,
    regardless of checkpoint correctness. Getting there also required fixing a real delaymgr
    starvation bug (checkpoint scheduled but never executed, commit `b9ad018`) and a real
    heap-use-after-free in `INVOKE_MSG_HANDLER` (`include/netio.h`) surfaced by the first-ever live
    exercise of a snapshot transfer landing during an election. With all fixes in place:
    `--force-snapshot` soak result = `committed_divergence=0 committed_job_loss=0 result=PASS`,
    target node's log confirms a real `HA: snapshot` install (not just ordinary WAL catch-up), and
    `verify_no_loss` confirms every submitted job (entity content, not raw WAL) is present and
    correct post-restore on the snapshot-caught-up member. Full 217-test controller suite green
    before/after; a 3-round `--chaotic` regression also clean._

- [x] 8. Validation: rolling-upgrade compatibility
- [x] 8.1 Verify a pre-feature checkpoint file is handled safely by the new binary in a live
      cluster
  - Plant a checkpoint file in the legacy (position-only) format, start a member on the upgraded
    binary against it, and observe its behavior
  - Observable: the member does not crash, logs the legacy-format fallback, and reaches full
    committed consistency through normal catch-up rather than reporting false-positive
    completeness
  - _Requirements: 5.1, 5.2_
  - _Depends: 6.1_
  - (8.1) `persist_recover`'s local boot-time snapshot load (persist.c ~2790) passed `NULL` for
    `out_entities_restored` -- it had no observability parity with the wire receive path's
    (`ha_handle_snapshot_chunk`, task 6.1) legacy-vs-entity-bearing INFO log. Added the same
    conditional INFO logging at this call site (small, additive, no behavior change -- WAL replay
    in Step 3 still applies unconditionally either way). Live-verified: planted a hand-crafted
    16-byte legacy sentinel (`XCESNAP\0` + LE checkpoint_lsn=1, no entity frames) as ctrl-1's own
    `snap-0000000000000001.capnp` in a 3-node cluster (reusing the ha-entity-snap soak deploy,
    WAL segments left intact), restarted all 3. ctrl-1: no crash, logged
    "persist_recover: ... is a legacy position-only checkpoint ... continuing via normal WAL
    replay", was elected primary, served the full 37-job list, and `walcompare` reported all 3
    members' committed prefixes (LSN 1..49) byte-identical -- confirms full committed consistency
    reached via ordinary WAL replay, no false-positive completeness.

## Implementation Notes
- (1.1) Test fixture `snapent_make`/`snapent_teardown` lives in `test_persist_snapshot_entities.c`; seeded jobs currently carry only jobid+state+an all-zeroed submit. Task 2.1's "field-identical round-trip" assertion needs richer submit-payload seeding (e.g. non-zero cmdline/user/queue in `xce_job_submit_t`) to be a meaningful field-by-field comparison — extend `snapent_seed_jobs`, do not re-seed separately.
- (2.1) `write_snapshot`'s existing `fdatasync(fd)` call (persist.c:1505) does not check its return value at all — task 2.2 must fix this while adding entity encoding, since design.md's error-handling contract ("any failure at any step -- encode, write, sync -- leaves no file at the final path") requires it.
- (2.1) The write-failure-discard test (`test_snapent_checkpoint_write_failure_leaves_no_final_file`) only exercises the pre-existing `open()`-failure branch (via directory removal) and passes today without any entity-encode loop existing yet. Task 2.2 should add a genuine mid-encode-loop fault-injection test (failing to encode e.g. the Nth of N jobs) once the per-job encode loop exists — the current test cannot validate that path.
- (2.2) `restore_job` (job.c) decodes via `decode_Job` → `decode_JobSubmitData_ptr`/`decode_JobStartInfo_ptr` (STRING_DUP'd fields) into a local `xce_job_t j`; that decoded struct is never released via `free_Job(&j)` after `create_job` takes its own copy — a real, pre-existing leak on `restore_job`'s decode path. Not touched by task 2.2 (its `load_snapshot` excursion was fully reverted, see next note) — task 5.2 will reintroduce this call site and must fix the leak there.
- (2.2) First attempt ALSO touched `load_snapshot`/`persist.h` to make task 2.1's round-trip tests pass end-to-end, but review found it advanced `current_lsn`/`checkpoint_lsn` BEFORE the entity-decode loop ran -- a live silent-divergence hazard reachable through the unmodified `ha_handle_snapshot_chunk` production path (reachable today, not hypothetical, since `write_snapshot` now always emits entity-bearing checkpoints). REMEDIATED by fully reverting `load_snapshot`/`persist.h` to pre-2.2 state (byte-identical to commit `0f344da`) and rewriting the round-trip tests to parse `write_snapshot`'s raw on-disk bytes directly (new `snapent_read_checkpoint_jobs` helper using `read_Job`/`decode_Job`) instead of round-tripping through `persist_load_snapshot`. Task 5.2 must implement `load_snapshot`'s entity-decode from scratch per design.md's restore-before-advance-position contract -- do not assume any decode scaffolding exists yet.
- (3.1) `snapent_read_checkpoint_queues`'s job-section skip loop duplicates `snapent_read_checkpoint_jobs`'s (task 2.2) skip logic verbatim rather than sharing a helper -- minor, non-blocking; consider factoring a shared "skip past job section" reader if task 5.x's test coverage grows the file further.
- (3.2) Pre-existing ASan leak in the test fixture itself (`snapent_make` -> `makeInstance`/`makeJob`/`makeQueue`/`xkvmgr_add`, ~18KB/113 allocations), unrelated to any production code -- present since task 1.1, masked by the project's standing `ASAN_OPTIONS=detect_leaks=0` convention for this suite. Not blocking any task; worth a dedicated fixture-teardown cleanup pass if this test file keeps growing.
- (4.1) `snapshot_inflight_lsn[peer_idx]` clears to 0 BEFORE `snapshot_inflight[peer_idx]` clears to 0 in every path (completion + all 3 failure sites in `ha_snapshot_send_dispatch`). Task 4.3's `rotate_snapshots` consultation must therefore treat `snapshot_inflight_lsn[i] != 0` as the "protected" signal directly (checkpoint LSNs are never 0), not "check flag==1 first, then trust LSN" -- there's a narrow window where the flag is still 1 but the LSN has already cleared.
- (execution note, after 4.1) Remaining tasks split by risk given a mid-session spend-limit hit: low-risk/small/isolated tasks (4.2, 4.3, 7.1, 8.1) executed directly in the main context with self-review (build + test + boundary/TBD/secrets grep), no subagent dispatch. High-risk tasks (5.1/5.2 `load_snapshot` restore -- the exact function that caused task 2.2's rejection; 6.1 wiring into the live `ha_handle_snapshot_chunk` production path) keep the full implementer + independent-reviewer subagent cycle, since that adversarial review already caught a real live-hazard bug once this spec (task 2.2) and the remaining risk is concentrated there, not spread evenly.
- (5.1) `persist_load_snapshot`'s wrapper currently maps `*out_entities_restored = (rc==0)` -- a deliberately naive stopgap so the 3 RED tests fail for the right reasons. `load_snapshot`'s real body is completely untouched (still watermark-only, byte-identical to before this task). Task 5.2 must replace the wrapper's naive mapping wholesale with real format-detection + restore-then-gate-position logic; it owns turning `load_snapshot` from a no-op-on-entities function into the real restore path.
- (5.2) `restore_queue_state`'s `-ENOENT` ("queue not yet built from config") aborts the ENTIRE checkpoint restore, including already-successfully-restored jobs earlier in the same file -- this is design-compliant (design.md/Req 2.3 are unconditional: any restore failure leaves position unchanged) and not a defect, but it is a real residual availability risk: a benign config-lag on the destination member (a queue the source references not yet locally provisioned) would make catch-up fail outright rather than skip-and-continue. WATCH-ITEM for task 7.1's chaotic soak: if a catch-up round gets stuck/never completes, this exact code path (queue.c's restore_queue_state ENOENT -> persist.c's load_snapshot_entities abort) is the first place to look.
- (5.2) Test-only, non-blocking: the destination fixture's `snapent_teardown` does not release job_t objects added via `restore_job` (only the ones seeded at fixture creation) -- a pre-existing category of test-fixture-teardown gap (same class as tasks.md's (3.2) note), now slightly enlarged since load-path tests exercise restore_job against a destination fixture. Not production code; extend snapent_teardown if this file keeps growing.