myxce/.kiro/specs/ha-entity-snapshots/tasks.md

Implementation Plan

• 1. Foundation: entity-snapshot test harness

• 1.1 Build a reusable test fixture for a persist_t backed by a populated job/queue table

  • Model on the existing make_hook_test_persist / cbrp_setup fixture patterns in the test suite
  • Fixture can seed N jobs (varying state) and M queues (varying status) into the cluster's job/queue tables
  • Observable: a new test file exists and compiles, exposing a helper that produces a ready persist_t+cluster_t with seeded entities
  • Requirements: 1.1, 2.1

• 2. Core: entity-bearing checkpoint write path — jobs

• 2.1 RED: job checkpoint content, consistency, and failure-discard tests

  • Assert a checkpoint containing N jobs round-trips through load with all N job records field-identical
  • Assert a job actively transitioning state on a concurrent thread during the checkpoint walk is captured as a single consistent pre- or post-transition value, never torn
  • Assert a simulated write/encode failure partway through leaves no file at the final checkpoint path — the same discard behavior that (per design) is what already makes the existing trim/rotate sequencing satisfy the retention-safety requirement
  • Observable: new tests compile and fail (RED) against the current sentinel-only write_snapshot
  • Requirements: 1.1, 1.2, 1.3, 3.1, 3.2, 3.3
  • Boundary: Persistence Layer (write_snapshot)

• 2.2 Collect and encode live jobs into the checkpoint under per-entity consistency

  • Enumerate live jobs under the existing retention policy, take each job's own read lock nested inside the table walk, encode via the existing job serializer, and append to the checkpoint after the current header
  • Any failure at any step discards the temp file and leaves no file at the final path
  • Observable: the RED tests from 2.1 pass
  • Requirements: 1.1, 1.2, 1.3, 1.4
  • Boundary: Persistence Layer (write_snapshot)

• 3. Core: entity-bearing checkpoint write path — queues

• 3.1 RED: queue admin-status round-trip test

  • Assert a queue's (name, status) pair written into the checkpoint and read back matches exactly
  • Observable: new test compiles and fails (RED)
  • Requirements: 1.1
  • Boundary: Persistence Layer (write_snapshot)
  • Depends: 2.2

• 3.2 Collect queue admin-status into the checkpoint

  • Enumerate every queue present in the live table (its table presence already implies config-provisioned existence), take each queue's own read lock nested inside the table walk, append its name+status as a plain record (no serialization schema needed — matches the existing restore function's exact contract)
  • Observable: the RED test from 3.1 passes; a checkpoint containing M queues restores M correct statuses
  • Requirements: 1.1, 1.2
  • Boundary: Persistence Layer (write_snapshot)

• 4. Core: protect an in-flight checkpoint transfer from rotation

• 4.1 (P) Track which checkpoint position each in-flight peer transfer references

  • Add a per-peer tracking array alongside the existing in-flight-transfer flag; set it when a transfer begins, clear it when the transfer completes
  • Observable: a unit test confirms the tracked position is set before a transfer's first byte goes out and cleared after completion
  • Requirements: 3.1, 3.2, 3.3
  • Boundary: HA Replication Layer

• 4.2 RED: rotation-guard test

  • Assert a checkpoint file whose position matches an in-flight transfer survives file rotation even when it would otherwise be rotated out
  • Observable: new test compiles and fails (RED) against the current rotation logic
  • Requirements: 3.1, 3.2, 3.3
  • Depends: 4.1

• 4.3 Skip rotating out a checkpoint file that is currently mid-transfer

  • Extend the existing file-rotation pass to check the in-flight tracking before deleting a candidate file
  • Observable: the RED test from 4.2 passes
  • Requirements: 3.1, 3.2, 3.3

• 5. Core: entity restoration on load

• 5.1 RED: restore-before-advance, failure-safety, and legacy-format tests

  • Assert loading a checkpoint with N jobs + M queues restores all of them before the member's position advances
  • Assert a deliberately corrupted entity record causes the load to fail with the member's position left unchanged
  • Assert a checkpoint file predating this feature (position marker only) is accepted without crashing, reports itself as position-only, and does not falsely claim full restoration
  • Observable: new tests compile and fail (RED) against the current watermark-only load
  • Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2
  • Boundary: Persistence Layer (load_snapshot)
  • Depends: 2.2, 3.2

• 5.2 Restore entities before advancing the member's position, with legacy-format detection

  • Detect whether a checkpoint carries entity content or is a legacy position-only file
  • Restore every job then every queue status via the existing restore functions, in file order; on any failure, stop and leave position unchanged
  • Only after every entity restores successfully, advance the member's committed position; extend the load function's contract with an indicator of whether entities were actually restored
  • Observable: the RED tests from 5.1 pass
  • Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2
  • Boundary: Persistence Layer (load_snapshot)

• 6. Integration: wire the catch-up receive path to the new restore contract

• 6.1 Consume the new restoration indicator in the snapshot-chunk receive handler

  • The handler advances the member's applied position only after the load succeeds (unchanged ordering); on a legacy (position-only) file it logs the fallback rather than treating it as a failure
  • Observable: given a legacy-format file, the member logs the fallback and continues through normal catch-up; given an entity-bearing file, the member's applied position advances only after restoration succeeds
  • Requirements: 2.4, 4.1, 4.2, 5.1, 5.2
  • Depends: 5.2

• 7. Validation: chaotic catch-up soak proves end-to-end correctness

• 7.1 Extend the chaotic HA failover soak to force at least one full-checkpoint catch-up round

  • Configure or trigger a round where a member rejoins empty or far enough behind that it must receive a full checkpoint rather than incremental history
  • Observable: the soak verdict for that round reports zero committed-job loss and zero committed-prefix divergence, and existing comparison tooling confirms the rejoined member's job/queue state matches the rest of the cluster
  • Requirements: 4.1, 4.2, 4.3

• 8. Validation: rolling-upgrade compatibility

• 8.1 Verify a pre-feature checkpoint file is handled safely by the new binary in a live cluster

  • Plant a checkpoint file in the legacy (position-only) format, start a member on the upgraded binary against it, and observe its behavior
  • Observable: the member does not crash, logs the legacy-format fallback, and reaches full committed consistency through normal catch-up rather than reporting false-positive completeness
  • Requirements: 5.1, 5.2
  • Depends: 6.1

Implementation Notes

• (1.1) Test fixture snapent_make/snapent_teardown lives in test_persist_snapshot_entities.c; seeded jobs currently carry only jobid+state+an all-zeroed submit. Task 2.1's "field-identical round-trip" assertion needs richer submit-payload seeding (e.g. non-zero cmdline/user/queue in xce_job_submit_t) to be a meaningful field-by-field comparison — extend snapent_seed_jobs, do not re-seed separately.
• (2.1) write_snapshot's existing fdatasync(fd) call (persist.c:1505) does not check its return value at all — task 2.2 must fix this while adding entity encoding, since design.md's error-handling contract ("any failure at any step -- encode, write, sync -- leaves no file at the final path") requires it.
• (2.1) The write-failure-discard test (test_snapent_checkpoint_write_failure_leaves_no_final_file) only exercises the pre-existing open()-failure branch (via directory removal) and passes today without any entity-encode loop existing yet. Task 2.2 should add a genuine mid-encode-loop fault-injection test (failing to encode e.g. the Nth of N jobs) once the per-job encode loop exists — the current test cannot validate that path.
• (2.2) restore_job (job.c) decodes via decode_Job → decode_JobSubmitData_ptr/decode_JobStartInfo_ptr (STRING_DUP'd fields) into a local xce_job_t j; that decoded struct is never released via free_Job(&j) after create_job takes its own copy — a real, pre-existing leak on restore_job's decode path. Not touched by task 2.2 (its load_snapshot excursion was fully reverted, see next note) — task 5.2 will reintroduce this call site and must fix the leak there.
• (2.2) First attempt ALSO touched load_snapshot/persist.h to make task 2.1's round-trip tests pass end-to-end, but review found it advanced current_lsn/checkpoint_lsn BEFORE the entity-decode loop ran -- a live silent-divergence hazard reachable through the unmodified ha_handle_snapshot_chunk production path (reachable today, not hypothetical, since write_snapshot now always emits entity-bearing checkpoints). REMEDIATED by fully reverting load_snapshot/persist.h to pre-2.2 state (byte-identical to commit 0f344da) and rewriting the round-trip tests to parse write_snapshot's raw on-disk bytes directly (new snapent_read_checkpoint_jobs helper using read_Job/decode_Job) instead of round-tripping through persist_load_snapshot. Task 5.2 must implement load_snapshot's entity-decode from scratch per design.md's restore-before-advance-position contract -- do not assume any decode scaffolding exists yet.
• (3.1) snapent_read_checkpoint_queues's job-section skip loop duplicates snapent_read_checkpoint_jobs's (task 2.2) skip logic verbatim rather than sharing a helper -- minor, non-blocking; consider factoring a shared "skip past job section" reader if task 5.x's test coverage grows the file further.