feat(ha-entity-snapshots): add spec + task 1.1 test fixture

Adds the ha-entity-snapshots Kiro spec (requirements, design, research,
tasks) for task 5.10a / raft-ha-core-safety: making the HA checkpoint an
actually-restorable copy of committed job/queue state instead of a
position-only sentinel, closing the residual chaotic-soak divergence gap.

Task 1.1 (foundation): reusable test fixture snapent_make/snapent_teardown
in a new test_persist_snapshot_entities.c, seeding N jobs (varying state)
and M queues (varying status) into a real persist_t+cluster_t for later
write/load/rotation-guard tests to build on. Build clean; new suite 1/1;
regression spot-checks (test_persist_hooks 19/19, test_commit_barrier_reply
22/22) clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX

.kiro/specs/ha-entity-snapshots/tasks.md

@@ -0,0 +1,127 @@
+# Implementation Plan
+
+- [x] 1. Foundation: entity-snapshot test harness
+- [x] 1.1 Build a reusable test fixture for a persist_t backed by a populated job/queue table
+  - Model on the existing `make_hook_test_persist` / `cbrp_setup` fixture patterns in the test
+    suite
+  - Fixture can seed N jobs (varying state) and M queues (varying status) into the cluster's
+    job/queue tables
+  - Observable: a new test file exists and compiles, exposing a helper that produces a ready
+    persist_t+cluster_t with seeded entities
+  - _Requirements: 1.1, 2.1_
+
+- [ ] 2. Core: entity-bearing checkpoint write path — jobs
+- [ ] 2.1 RED: job checkpoint content, consistency, and failure-discard tests
+  - Assert a checkpoint containing N jobs round-trips through load with all N job records
+    field-identical
+  - Assert a job actively transitioning state on a concurrent thread during the checkpoint walk
+    is captured as a single consistent pre- or post-transition value, never torn
+  - Assert a simulated write/encode failure partway through leaves no file at the final
+    checkpoint path — the same discard behavior that (per design) is what already makes the
+    existing trim/rotate sequencing satisfy the retention-safety requirement
+  - Observable: new tests compile and fail (RED) against the current sentinel-only write_snapshot
+  - _Requirements: 1.1, 1.2, 1.3, 3.1, 3.2, 3.3_
+  - _Boundary: Persistence Layer (write_snapshot)_
+- [ ] 2.2 Collect and encode live jobs into the checkpoint under per-entity consistency
+  - Enumerate live jobs under the existing retention policy, take each job's own read lock
+    nested inside the table walk, encode via the existing job serializer, and append to the
+    checkpoint after the current header
+  - Any failure at any step discards the temp file and leaves no file at the final path
+  - Observable: the RED tests from 2.1 pass
+  - _Requirements: 1.1, 1.2, 1.3, 1.4_
+  - _Boundary: Persistence Layer (write_snapshot)_
+
+- [ ] 3. Core: entity-bearing checkpoint write path — queues
+- [ ] 3.1 RED: queue admin-status round-trip test
+  - Assert a queue's (name, status) pair written into the checkpoint and read back matches
+    exactly
+  - Observable: new test compiles and fails (RED)
+  - _Requirements: 1.1_
+  - _Boundary: Persistence Layer (write_snapshot)_
+  - _Depends: 2.2_
+- [ ] 3.2 Collect queue admin-status into the checkpoint
+  - Enumerate every queue present in the live table (its table presence already implies
+    config-provisioned existence), take each queue's own read lock nested inside the table walk,
+    append its name+status as a plain record (no serialization schema needed — matches the
+    existing restore function's exact contract)
+  - Observable: the RED test from 3.1 passes; a checkpoint containing M queues restores M
+    correct statuses
+  - _Requirements: 1.1, 1.2_
+  - _Boundary: Persistence Layer (write_snapshot)_
+
+- [ ] 4. Core: protect an in-flight checkpoint transfer from rotation
+- [ ] 4.1 (P) Track which checkpoint position each in-flight peer transfer references
+  - Add a per-peer tracking array alongside the existing in-flight-transfer flag; set it when a
+    transfer begins, clear it when the transfer completes
+  - Observable: a unit test confirms the tracked position is set before a transfer's first byte
+    goes out and cleared after completion
+  - _Requirements: 3.1, 3.2, 3.3_
+  - _Boundary: HA Replication Layer_
+- [ ] 4.2 RED: rotation-guard test
+  - Assert a checkpoint file whose position matches an in-flight transfer survives file rotation
+    even when it would otherwise be rotated out
+  - Observable: new test compiles and fails (RED) against the current rotation logic
+  - _Requirements: 3.1, 3.2, 3.3_
+  - _Depends: 4.1_
+- [ ] 4.3 Skip rotating out a checkpoint file that is currently mid-transfer
+  - Extend the existing file-rotation pass to check the in-flight tracking before deleting a
+    candidate file
+  - Observable: the RED test from 4.2 passes
+  - _Requirements: 3.1, 3.2, 3.3_
+
+- [ ] 5. Core: entity restoration on load
+- [ ] 5.1 RED: restore-before-advance, failure-safety, and legacy-format tests
+  - Assert loading a checkpoint with N jobs + M queues restores all of them before the member's
+    position advances
+  - Assert a deliberately corrupted entity record causes the load to fail with the member's
+    position left unchanged
+  - Assert a checkpoint file predating this feature (position marker only) is accepted without
+    crashing, reports itself as position-only, and does not falsely claim full restoration
+  - Observable: new tests compile and fail (RED) against the current watermark-only load
+  - _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2_
+  - _Boundary: Persistence Layer (load_snapshot)_
+  - _Depends: 2.2, 3.2_
+- [ ] 5.2 Restore entities before advancing the member's position, with legacy-format detection
+  - Detect whether a checkpoint carries entity content or is a legacy position-only file
+  - Restore every job then every queue status via the existing restore functions, in file order;
+    on any failure, stop and leave position unchanged
+  - Only after every entity restores successfully, advance the member's committed position;
+    extend the load function's contract with an indicator of whether entities were actually
+    restored
+  - Observable: the RED tests from 5.1 pass
+  - _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1, 5.2_
+  - _Boundary: Persistence Layer (load_snapshot)_
+
+- [ ] 6. Integration: wire the catch-up receive path to the new restore contract
+- [ ] 6.1 Consume the new restoration indicator in the snapshot-chunk receive handler
+  - The handler advances the member's applied position only after the load succeeds (unchanged
+    ordering); on a legacy (position-only) file it logs the fallback rather than treating it as
+    a failure
+  - Observable: given a legacy-format file, the member logs the fallback and continues through
+    normal catch-up; given an entity-bearing file, the member's applied position advances only
+    after restoration succeeds
+  - _Requirements: 2.4, 4.1, 4.2, 5.1, 5.2_
+  - _Depends: 5.2_
+
+- [ ] 7. Validation: chaotic catch-up soak proves end-to-end correctness
+- [ ] 7.1 Extend the chaotic HA failover soak to force at least one full-checkpoint catch-up round
+  - Configure or trigger a round where a member rejoins empty or far enough behind that it must
+    receive a full checkpoint rather than incremental history
+  - Observable: the soak verdict for that round reports zero committed-job loss and zero
+    committed-prefix divergence, and existing comparison tooling confirms the rejoined member's
+    job/queue state matches the rest of the cluster
+  - _Requirements: 4.1, 4.2, 4.3_
+
+- [ ] 8. Validation: rolling-upgrade compatibility
+- [ ] 8.1 Verify a pre-feature checkpoint file is handled safely by the new binary in a live
+      cluster
+  - Plant a checkpoint file in the legacy (position-only) format, start a member on the upgraded
+    binary against it, and observe its behavior
+  - Observable: the member does not crash, logs the legacy-format fallback, and reaches full
+    committed consistency through normal catch-up rather than reporting false-positive
+    completeness
+  - _Requirements: 5.1, 5.2_
+  - _Depends: 6.1_
+
+## Implementation Notes
+- (1.1) Test fixture `snapent_make`/`snapent_teardown` lives in `test_persist_snapshot_entities.c`; seeded jobs currently carry only jobid+state+an all-zeroed submit. Task 2.1's "field-identical round-trip" assertion needs richer submit-payload seeding (e.g. non-zero cmdline/user/queue in `xce_job_submit_t`) to be a meaningful field-by-field comparison — extend `snapent_seed_jobs`, do not re-seed separately.