myxce/.kiro/specs/ha-entity-snapshots/tasks.md

Implementation Plan

• 1. Foundation: entity-snapshot test harness

• 1.1 Build a reusable test fixture for a persist_t backed by a populated job/queue table

  • Model on the existing make_hook_test_persist / cbrp_setup fixture patterns in the test suite
  • Fixture can seed N jobs (varying state) and M queues (varying status) into the cluster's job/queue tables
  • Observable: a new test file exists and compiles, exposing a helper that produces a ready persist_t+cluster_t with seeded entities
  • Requirements: 1.1, 2.1

• 2. Core: entity-bearing checkpoint write path — jobs

• 2.1 RED: job checkpoint content, consistency, and failure-discard tests

  • Assert a checkpoint containing N jobs round-trips through load with all N job records field-identical
  • Assert a job actively transitioning state on a concurrent thread during the checkpoint walk is captured as a single consistent pre- or post-transition value, never torn
  • Assert a simulated write/encode failure partway through leaves no file at the final checkpoint path — the same discard behavior that (per design) is what already makes the existing trim/rotate sequencing satisfy the retention-safety requirement
  • Observable: new tests compile and fail (RED) against the current sentinel-only write_snapshot
  • Requirements: 1.1, 1.2, 1.3, 3.1, 3.2, 3.3
  • Boundary: Persistence Layer (write_snapshot)

• 2.2 Collect and encode live jobs into the checkpoint under per-entity consistency

  • Enumerate live jobs under the existing retention policy, take each job's own read lock nested inside the table walk, encode via the existing job serializer, and append to the checkpoint after the current header
  • Any failure at any step discards the temp file and leaves no file at the final path
  • Observable: the RED tests from 2.1 pass
  • Requirements: 1.1, 1.2, 1.3, 1.4
  • Boundary: Persistence Layer (write_snapshot)

• 3. Core: entity-bearing checkpoint write path — queues

• 3.1 RED: queue admin-status round-trip test

  • Assert a queue's (name, status) pair written into the checkpoint and read back matches exactly
  • Observable: new test compiles and fails (RED)
  • Requirements: 1.1
  • Boundary: Persistence Layer (write_snapshot)
  • Depends: 2.2

• 3.2 Collect queue admin-status into the checkpoint

  • Enumerate every queue present in the live table (its table presence already implies config-provisioned existence), take each queue's own read lock nested inside the table walk, append its name+status as a plain record (no serialization schema needed — matches the existing restore function's exact contract)
  • Observable: the RED test from 3.1 passes; a checkpoint containing M queues restores M correct statuses
  • Requirements: 1.1, 1.2
  • Boundary: Persistence Layer (write_snapshot)

• 4. Core: protect an in-flight checkpoint transfer from rotation

• 4.1 (P) Track which checkpoint position each in-flight peer transfer references

  • Add a per-peer tracking array alongside the existing in-flight-transfer flag; set it when a transfer begins, clear it when the transfer completes
  • Observable: a unit test confirms the tracked position is set before a transfer's first byte goes out and cleared after completion
  • Requirements: 3.1, 3.2, 3.3
  • Boundary: HA Replication Layer

• 4.2 RED: rotation-guard test

  • Assert a checkpoint file whose position matches an in-flight transfer survives file rotation even when it would otherwise be rotated out
  • Observable: new test compiles and fails (RED) against the current rotation logic
  • Requirements: 3.1, 3.2, 3.3
  • Depends: 4.1

• 4.3 Skip rotating out a checkpoint file that is currently mid-transfer

  • Extend the existing file-rotation pass to check the in-flight tracking before deleting a candidate file
  • Observable: the RED test from 4.2 passes
  • Requirements: 3.1, 3.2, 3.3

• 5. Core: entity restoration on load

• 5.1 RED: restore-before-advance, failure-safety, and legacy-format tests

  • Assert loading a checkpoint with N jobs + M queues restores all of them before the member's position advances
  • Assert a deliberately corrupted entity record causes the load to fail with the member's position left unchanged
  • Assert a checkpoint file predating this feature (position marker only) is accepted without crashing, reports itself as position-only, and does not falsely claim full restoration
  • Observable: new tests compile and fail (RED) against the current watermark-only load
  • Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2
  • Boundary: Persistence Layer (load_snapshot)
  • Depends: 2.2, 3.2

• 5.2 Restore entities before advancing the member's position, with legacy-format detection

  • Detect whether a checkpoint carries entity content or is a legacy position-only file
  • Restore every job then every queue status via the existing restore functions, in file order; on any failure, stop and leave position unchanged
  • Only after every entity restores successfully, advance the member's committed position; extend the load function's contract with an indicator of whether entities were actually restored
  • Observable: the RED tests from 5.1 pass
  • Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2
  • Boundary: Persistence Layer (load_snapshot)

• 6. Integration: wire the catch-up receive path to the new restore contract

• 6.1 Consume the new restoration indicator in the snapshot-chunk receive handler

  • The handler advances the member's applied position only after the load succeeds (unchanged ordering); on a legacy (position-only) file it logs the fallback rather than treating it as a failure
  • Observable: given a legacy-format file, the member logs the fallback and continues through normal catch-up; given an entity-bearing file, the member's applied position advances only after restoration succeeds
  • Requirements: 2.4, 4.1, 4.2, 5.1, 5.2
  • Depends: 5.2

• 7. Validation: chaotic catch-up soak proves end-to-end correctness

• 7.1 Extend the chaotic HA failover soak to force at least one full-checkpoint catch-up round

  • Configure or trigger a round where a member rejoins empty or far enough behind that it must receive a full checkpoint rather than incremental history
  • Observable: the soak verdict for that round reports zero committed-job loss and zero committed-prefix divergence, and existing comparison tooling confirms the rejoined member's job/queue state matches the rest of the cluster
  • Requirements: 4.1, 4.2, 4.3
  • RESOLVED 2026-07-01 (raft-ha-core-safety task 5.10a, same investigation — see that spec's task 5.10 note for the full chain). The blocking walcompare per-member checkpoint-floor gap was fixed (commit f3c86a6). The "specific soak run never actually exercised a full-checkpoint catch-up at all" gap was closed by building a --force-snapshot soak mode (commit 9f4292d) plus a --wal-segment-bytes CLI override (this session) — the WAL segment physical size was a hardcoded 64 MB compile-time constant, and trim_wal_segments() never deletes the currently- open segment, so at soak job counts a segment never rotated and nothing was ever trimmable, regardless of checkpoint correctness. Getting there also required fixing a real delaymgr starvation bug (checkpoint scheduled but never executed, commit b9ad018) and a real heap-use-after-free in INVOKE_MSG_HANDLER (include/netio.h) surfaced by the first-ever live exercise of a snapshot transfer landing during an election. With all fixes in place: --force-snapshot soak result = committed_divergence=0 committed_job_loss=0 result=PASS, target node's log confirms a real HA: snapshot install (not just ordinary WAL catch-up), and verify_no_loss confirms every submitted job (entity content, not raw WAL) is present and correct post-restore on the snapshot-caught-up member. Full 217-test controller suite green before/after; a 3-round --chaotic regression also clean.

• 8. Validation: rolling-upgrade compatibility

• 8.1 Verify a pre-feature checkpoint file is handled safely by the new binary in a live cluster

  • Plant a checkpoint file in the legacy (position-only) format, start a member on the upgraded binary against it, and observe its behavior
  • Observable: the member does not crash, logs the legacy-format fallback, and reaches full committed consistency through normal catch-up rather than reporting false-positive completeness
  • Requirements: 5.1, 5.2
  • Depends: 6.1
  • (8.1) persist_recover's local boot-time snapshot load (persist.c ~2790) passed NULL for out_entities_restored -- it had no observability parity with the wire receive path's (ha_handle_snapshot_chunk, task 6.1) legacy-vs-entity-bearing INFO log. Added the same conditional INFO logging at this call site (small, additive, no behavior change -- WAL replay in Step 3 still applies unconditionally either way). Live-verified: planted a hand-crafted 16-byte legacy sentinel (XCESNAP\0 + LE checkpoint_lsn=1, no entity frames) as ctrl-1's own snap-0000000000000001.capnp in a 3-node cluster (reusing the ha-entity-snap soak deploy, WAL segments left intact), restarted all 3. ctrl-1: no crash, logged "persist_recover: ... is a legacy position-only checkpoint ... continuing via normal WAL replay", was elected primary, served the full 37-job list, and walcompare reported all 3 members' committed prefixes (LSN 1..49) byte-identical -- confirms full committed consistency reached via ordinary WAL replay, no false-positive completeness.

Implementation Notes

• (1.1) Test fixture snapent_make/snapent_teardown lives in test_persist_snapshot_entities.c; seeded jobs currently carry only jobid+state+an all-zeroed submit. Task 2.1's "field-identical round-trip" assertion needs richer submit-payload seeding (e.g. non-zero cmdline/user/queue in xce_job_submit_t) to be a meaningful field-by-field comparison — extend snapent_seed_jobs, do not re-seed separately.
• (2.1) write_snapshot's existing fdatasync(fd) call (persist.c:1505) does not check its return value at all — task 2.2 must fix this while adding entity encoding, since design.md's error-handling contract ("any failure at any step -- encode, write, sync -- leaves no file at the final path") requires it.
• (2.1) The write-failure-discard test (test_snapent_checkpoint_write_failure_leaves_no_final_file) only exercises the pre-existing open()-failure branch (via directory removal) and passes today without any entity-encode loop existing yet. Task 2.2 should add a genuine mid-encode-loop fault-injection test (failing to encode e.g. the Nth of N jobs) once the per-job encode loop exists — the current test cannot validate that path.
• (2.2) restore_job (job.c) decodes via decode_Job → decode_JobSubmitData_ptr/decode_JobStartInfo_ptr (STRING_DUP'd fields) into a local xce_job_t j; that decoded struct is never released via free_Job(&j) after create_job takes its own copy — a real, pre-existing leak on restore_job's decode path. Not touched by task 2.2 (its load_snapshot excursion was fully reverted, see next note) — task 5.2 will reintroduce this call site and must fix the leak there.
• (2.2) First attempt ALSO touched load_snapshot/persist.h to make task 2.1's round-trip tests pass end-to-end, but review found it advanced current_lsn/checkpoint_lsn BEFORE the entity-decode loop ran -- a live silent-divergence hazard reachable through the unmodified ha_handle_snapshot_chunk production path (reachable today, not hypothetical, since write_snapshot now always emits entity-bearing checkpoints). REMEDIATED by fully reverting load_snapshot/persist.h to pre-2.2 state (byte-identical to commit 0f344da) and rewriting the round-trip tests to parse write_snapshot's raw on-disk bytes directly (new snapent_read_checkpoint_jobs helper using read_Job/decode_Job) instead of round-tripping through persist_load_snapshot. Task 5.2 must implement load_snapshot's entity-decode from scratch per design.md's restore-before-advance-position contract -- do not assume any decode scaffolding exists yet.
• (3.1) snapent_read_checkpoint_queues's job-section skip loop duplicates snapent_read_checkpoint_jobs's (task 2.2) skip logic verbatim rather than sharing a helper -- minor, non-blocking; consider factoring a shared "skip past job section" reader if task 5.x's test coverage grows the file further.
• (3.2) Pre-existing ASan leak in the test fixture itself (snapent_make -> makeInstance/makeJob/makeQueue/xkvmgr_add, ~18KB/113 allocations), unrelated to any production code -- present since task 1.1, masked by the project's standing ASAN_OPTIONS=detect_leaks=0 convention for this suite. Not blocking any task; worth a dedicated fixture-teardown cleanup pass if this test file keeps growing.
• (4.1) snapshot_inflight_lsn[peer_idx] clears to 0 BEFORE snapshot_inflight[peer_idx] clears to 0 in every path (completion + all 3 failure sites in ha_snapshot_send_dispatch). Task 4.3's rotate_snapshots consultation must therefore treat snapshot_inflight_lsn[i] != 0 as the "protected" signal directly (checkpoint LSNs are never 0), not "check flag==1 first, then trust LSN" -- there's a narrow window where the flag is still 1 but the LSN has already cleared.
• (execution note, after 4.1) Remaining tasks split by risk given a mid-session spend-limit hit: low-risk/small/isolated tasks (4.2, 4.3, 7.1, 8.1) executed directly in the main context with self-review (build + test + boundary/TBD/secrets grep), no subagent dispatch. High-risk tasks (5.1/5.2 load_snapshot restore -- the exact function that caused task 2.2's rejection; 6.1 wiring into the live ha_handle_snapshot_chunk production path) keep the full implementer + independent-reviewer subagent cycle, since that adversarial review already caught a real live-hazard bug once this spec (task 2.2) and the remaining risk is concentrated there, not spread evenly.
• (5.1) persist_load_snapshot's wrapper currently maps *out_entities_restored = (rc==0) -- a deliberately naive stopgap so the 3 RED tests fail for the right reasons. load_snapshot's real body is completely untouched (still watermark-only, byte-identical to before this task). Task 5.2 must replace the wrapper's naive mapping wholesale with real format-detection + restore-then-gate-position logic; it owns turning load_snapshot from a no-op-on-entities function into the real restore path.
• (5.2) restore_queue_state's -ENOENT ("queue not yet built from config") aborts the ENTIRE checkpoint restore, including already-successfully-restored jobs earlier in the same file -- this is design-compliant (design.md/Req 2.3 are unconditional: any restore failure leaves position unchanged) and not a defect, but it is a real residual availability risk: a benign config-lag on the destination member (a queue the source references not yet locally provisioned) would make catch-up fail outright rather than skip-and-continue. WATCH-ITEM for task 7.1's chaotic soak: if a catch-up round gets stuck/never completes, this exact code path (queue.c's restore_queue_state ENOENT -> persist.c's load_snapshot_entities abort) is the first place to look.
• (5.2) Test-only, non-blocking: the destination fixture's snapent_teardown does not release job_t objects added via restore_job (only the ones seeded at fixture creation) -- a pre-existing category of test-fixture-teardown gap (same class as tasks.md's (3.2) note), now slightly enlarged since load-path tests exercise restore_job against a destination fixture. Not production code; extend snapent_teardown if this file keeps growing.