myxce/.kiro/specs/ha-entity-snapshots/tasks.md

# Implementation Plan

- [x] 1. Foundation: entity-snapshot test harness
- [x] 1.1 Build a reusable test fixture for a persist_t backed by a populated job/queue table
  - Model on the existing `make_hook_test_persist` / `cbrp_setup` fixture patterns in the test
    suite
  - Fixture can seed N jobs (varying state) and M queues (varying status) into the cluster's
    job/queue tables
  - Observable: a new test file exists and compiles, exposing a helper that produces a ready
    persist_t+cluster_t with seeded entities
  - _Requirements: 1.1, 2.1_

- [x] 2. Core: entity-bearing checkpoint write path — jobs
- [x] 2.1 RED: job checkpoint content, consistency, and failure-discard tests
  - Assert a checkpoint containing N jobs round-trips through load with all N job records
    field-identical
  - Assert a job actively transitioning state on a concurrent thread during the checkpoint walk
    is captured as a single consistent pre- or post-transition value, never torn
  - Assert a simulated write/encode failure partway through leaves no file at the final
    checkpoint path — the same discard behavior that (per design) is what already makes the
    existing trim/rotate sequencing satisfy the retention-safety requirement
  - Observable: new tests compile and fail (RED) against the current sentinel-only write_snapshot
  - _Requirements: 1.1, 1.2, 1.3, 3.1, 3.2, 3.3_
  - _Boundary: Persistence Layer (write_snapshot)_
- [x] 2.2 Collect and encode live jobs into the checkpoint under per-entity consistency
  - Enumerate live jobs under the existing retention policy, take each job's own read lock
    nested inside the table walk, encode via the existing job serializer, and append to the
    checkpoint after the current header
  - Any failure at any step discards the temp file and leaves no file at the final path
  - Observable: the RED tests from 2.1 pass
  - _Requirements: 1.1, 1.2, 1.3, 1.4_
  - _Boundary: Persistence Layer (write_snapshot)_

- [x] 3. Core: entity-bearing checkpoint write path — queues
- [x] 3.1 RED: queue admin-status round-trip test
  - Assert a queue's (name, status) pair written into the checkpoint and read back matches
    exactly
  - Observable: new test compiles and fails (RED)
  - _Requirements: 1.1_
  - _Boundary: Persistence Layer (write_snapshot)_
  - _Depends: 2.2_
- [x] 3.2 Collect queue admin-status into the checkpoint
  - Enumerate every queue present in the live table (its table presence already implies
    config-provisioned existence), take each queue's own read lock nested inside the table walk,
    append its name+status as a plain record (no serialization schema needed — matches the
    existing restore function's exact contract)
  - Observable: the RED test from 3.1 passes; a checkpoint containing M queues restores M
    correct statuses
  - _Requirements: 1.1, 1.2_
  - _Boundary: Persistence Layer (write_snapshot)_

- [x] 4. Core: protect an in-flight checkpoint transfer from rotation
- [x] 4.1 (P) Track which checkpoint position each in-flight peer transfer references
  - Add a per-peer tracking array alongside the existing in-flight-transfer flag; set it when a
    transfer begins, clear it when the transfer completes
  - Observable: a unit test confirms the tracked position is set before a transfer's first byte
    goes out and cleared after completion
  - _Requirements: 3.1, 3.2, 3.3_
  - _Boundary: HA Replication Layer_
- [x] 4.2 RED: rotation-guard test
  - Assert a checkpoint file whose position matches an in-flight transfer survives file rotation
    even when it would otherwise be rotated out
  - Observable: new test compiles and fails (RED) against the current rotation logic
  - _Requirements: 3.1, 3.2, 3.3_
  - _Depends: 4.1_
- [x] 4.3 Skip rotating out a checkpoint file that is currently mid-transfer
  - Extend the existing file-rotation pass to check the in-flight tracking before deleting a
    candidate file
  - Observable: the RED test from 4.2 passes
  - _Requirements: 3.1, 3.2, 3.3_

- [ ] 5. Core: entity restoration on load
- [x] 5.1 RED: restore-before-advance, failure-safety, and legacy-format tests
  - Assert loading a checkpoint with N jobs + M queues restores all of them before the member's
    position advances
  - Assert a deliberately corrupted entity record causes the load to fail with the member's
    position left unchanged
  - Assert a checkpoint file predating this feature (position marker only) is accepted without
    crashing, reports itself as position-only, and does not falsely claim full restoration
  - Observable: new tests compile and fail (RED) against the current watermark-only load
  - _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2_
  - _Boundary: Persistence Layer (load_snapshot)_
  - _Depends: 2.2, 3.2_
- [ ] 5.2 Restore entities before advancing the member's position, with legacy-format detection
  - Detect whether a checkpoint carries entity content or is a legacy position-only file
  - Restore every job then every queue status via the existing restore functions, in file order;
    on any failure, stop and leave position unchanged
  - Only after every entity restores successfully, advance the member's committed position;
    extend the load function's contract with an indicator of whether entities were actually
    restored
  - Observable: the RED tests from 5.1 pass
  - _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2_
  - _Boundary: Persistence Layer (load_snapshot)_

- [ ] 6. Integration: wire the catch-up receive path to the new restore contract
- [ ] 6.1 Consume the new restoration indicator in the snapshot-chunk receive handler
  - The handler advances the member's applied position only after the load succeeds (unchanged
    ordering); on a legacy (position-only) file it logs the fallback rather than treating it as
    a failure
  - Observable: given a legacy-format file, the member logs the fallback and continues through
    normal catch-up; given an entity-bearing file, the member's applied position advances only
    after restoration succeeds
  - _Requirements: 2.4, 4.1, 4.2, 5.1, 5.2_
  - _Depends: 5.2_

- [ ] 7. Validation: chaotic catch-up soak proves end-to-end correctness
- [ ] 7.1 Extend the chaotic HA failover soak to force at least one full-checkpoint catch-up round
  - Configure or trigger a round where a member rejoins empty or far enough behind that it must
    receive a full checkpoint rather than incremental history
  - Observable: the soak verdict for that round reports zero committed-job loss and zero
    committed-prefix divergence, and existing comparison tooling confirms the rejoined member's
    job/queue state matches the rest of the cluster
  - _Requirements: 4.1, 4.2, 4.3_

- [ ] 8. Validation: rolling-upgrade compatibility
- [ ] 8.1 Verify a pre-feature checkpoint file is handled safely by the new binary in a live
      cluster
  - Plant a checkpoint file in the legacy (position-only) format, start a member on the upgraded
    binary against it, and observe its behavior
  - Observable: the member does not crash, logs the legacy-format fallback, and reaches full
    committed consistency through normal catch-up rather than reporting false-positive
    completeness
  - _Requirements: 5.1, 5.2_
  - _Depends: 6.1_

## Implementation Notes
- (1.1) Test fixture `snapent_make`/`snapent_teardown` lives in `test_persist_snapshot_entities.c`; seeded jobs currently carry only jobid+state+an all-zeroed submit. Task 2.1's "field-identical round-trip" assertion needs richer submit-payload seeding (e.g. non-zero cmdline/user/queue in `xce_job_submit_t`) to be a meaningful field-by-field comparison — extend `snapent_seed_jobs`, do not re-seed separately.
- (2.1) `write_snapshot`'s existing `fdatasync(fd)` call (persist.c:1505) does not check its return value at all — task 2.2 must fix this while adding entity encoding, since design.md's error-handling contract ("any failure at any step -- encode, write, sync -- leaves no file at the final path") requires it.
- (2.1) The write-failure-discard test (`test_snapent_checkpoint_write_failure_leaves_no_final_file`) only exercises the pre-existing `open()`-failure branch (via directory removal) and passes today without any entity-encode loop existing yet. Task 2.2 should add a genuine mid-encode-loop fault-injection test (failing to encode e.g. the Nth of N jobs) once the per-job encode loop exists — the current test cannot validate that path.
- (2.2) `restore_job` (job.c) decodes via `decode_Job` → `decode_JobSubmitData_ptr`/`decode_JobStartInfo_ptr` (STRING_DUP'd fields) into a local `xce_job_t j`; that decoded struct is never released via `free_Job(&j)` after `create_job` takes its own copy — a real, pre-existing leak on `restore_job`'s decode path. Not touched by task 2.2 (its `load_snapshot` excursion was fully reverted, see next note) — task 5.2 will reintroduce this call site and must fix the leak there.
- (2.2) First attempt ALSO touched `load_snapshot`/`persist.h` to make task 2.1's round-trip tests pass end-to-end, but review found it advanced `current_lsn`/`checkpoint_lsn` BEFORE the entity-decode loop ran -- a live silent-divergence hazard reachable through the unmodified `ha_handle_snapshot_chunk` production path (reachable today, not hypothetical, since `write_snapshot` now always emits entity-bearing checkpoints). REMEDIATED by fully reverting `load_snapshot`/`persist.h` to pre-2.2 state (byte-identical to commit `0f344da`) and rewriting the round-trip tests to parse `write_snapshot`'s raw on-disk bytes directly (new `snapent_read_checkpoint_jobs` helper using `read_Job`/`decode_Job`) instead of round-tripping through `persist_load_snapshot`. Task 5.2 must implement `load_snapshot`'s entity-decode from scratch per design.md's restore-before-advance-position contract -- do not assume any decode scaffolding exists yet.
- (3.1) `snapent_read_checkpoint_queues`'s job-section skip loop duplicates `snapent_read_checkpoint_jobs`'s (task 2.2) skip logic verbatim rather than sharing a helper -- minor, non-blocking; consider factoring a shared "skip past job section" reader if task 5.x's test coverage grows the file further.
- (3.2) Pre-existing ASan leak in the test fixture itself (`snapent_make` -> `makeInstance`/`makeJob`/`makeQueue`/`xkvmgr_add`, ~18KB/113 allocations), unrelated to any production code -- present since task 1.1, masked by the project's standing `ASAN_OPTIONS=detect_leaks=0` convention for this suite. Not blocking any task; worth a dedicated fixture-teardown cleanup pass if this test file keeps growing.
- (4.1) `snapshot_inflight_lsn[peer_idx]` clears to 0 BEFORE `snapshot_inflight[peer_idx]` clears to 0 in every path (completion + all 3 failure sites in `ha_snapshot_send_dispatch`). Task 4.3's `rotate_snapshots` consultation must therefore treat `snapshot_inflight_lsn[i] != 0` as the "protected" signal directly (checkpoint LSNs are never 0), not "check flag==1 first, then trust LSN" -- there's a narrow window where the flag is still 1 but the LSN has already cleared.
- (execution note, after 4.1) Remaining tasks split by risk given a mid-session spend-limit hit: low-risk/small/isolated tasks (4.2, 4.3, 7.1, 8.1) executed directly in the main context with self-review (build + test + boundary/TBD/secrets grep), no subagent dispatch. High-risk tasks (5.1/5.2 `load_snapshot` restore -- the exact function that caused task 2.2's rejection; 6.1 wiring into the live `ha_handle_snapshot_chunk` production path) keep the full implementer + independent-reviewer subagent cycle, since that adversarial review already caught a real live-hazard bug once this spec (task 2.2) and the remaining risk is concentrated there, not spread evenly.
- (5.1) `persist_load_snapshot`'s wrapper currently maps `*out_entities_restored = (rc==0)` -- a deliberately naive stopgap so the 3 RED tests fail for the right reasons. `load_snapshot`'s real body is completely untouched (still watermark-only, byte-identical to before this task). Task 5.2 must replace the wrapper's naive mapping wholesale with real format-detection + restore-then-gate-position logic; it owns turning `load_snapshot` from a no-op-on-entities function into the real restore path.