feat(config-upgrade-snapshot): define HostAdminState/QueueAdminState capnp structs, bump snapshot format version
Adds independently-extensible Cap'n Proto frames for the upgrade snapshot's host/queue admin-state sections (frame-only, not reusing Worker/QueueConf which bundle dynamic state) and bumps SNAPSHOT_FORMAT_VERSION 1->2 so a checkpoint written under the prior raw queue-frame layout is fail-loud rejected rather than misinterpreted. Task 1 of 19. Requirements: 1.9, 5.7
This commit is contained in:
parent
e0c5e8b7a4
commit
9e68335aee
6 changed files with 322 additions and 8 deletions
154
.kiro/specs/config-upgrade-snapshot/tasks.md
Normal file
154
.kiro/specs/config-upgrade-snapshot/tasks.md
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
# Implementation Plan
|
||||
|
||||
- [x] 1. Define HostAdminState and QueueAdminState Cap'n Proto structs and bump the snapshot format version
|
||||
- Add `HostAdminState { name; status }` and `QueueAdminState { name; status }` messages to the controller schema, independently extensible for future admin-state fields
|
||||
- Bump the on-disk checkpoint format version so any binary reading a checkpoint under the old queue-frame layout hard-rejects it rather than misinterpreting the bytes
|
||||
- Observable: schema compiles, generates the new message types, and a checkpoint file written under the new version is rejected by code still expecting the old version
|
||||
- _Requirements: 1.9, 5.7_
|
||||
|
||||
- [ ] 2. Core: snapshot entity and configuration payload — collection and write side
|
||||
- [ ] 2.1 Collect host admin-state frames into the checkpoint
|
||||
- Walk the live host table the same way existing job/queue frame collection does, encoding only administratively-set status bits (open/closed/locked/tagged) per host as a HostAdminState frame
|
||||
- Exclude every worker-reported, dynamically-changing field (resource values, liveness timestamps, non-admin status bits) from the encoded frame
|
||||
- Observable: a checkpoint taken while a host is administratively closed contains exactly one HostAdminState frame for that host with only the admin bits set, and none of its dynamic fields
|
||||
- _Requirements: 1.6, 1.7, 1.9_
|
||||
|
||||
- [ ] 2.2 Migrate queue admin-state frames to the new wire format
|
||||
- Replace the queue frame's fixed raw byte layout with QueueAdminState encoding, preserving the same name+status content
|
||||
- Observable: a checkpoint's queue section decodes via the new message type, and a checkpoint written under the prior raw layout is rejected fail-loud rather than silently reinterpreted
|
||||
- _Requirements: 1.9, 5.7_
|
||||
|
||||
- [ ] 2.3 Embed the active configuration bytes into every checkpoint
|
||||
- Read the locally-held active configuration bytes directly from where they are already durably saved on disk, and append them as a new checkpoint section after the host frames
|
||||
- Ensure this section is populated on every checkpoint (not only a marker-triggered one), so a member joining or catching up after the marker-triggered snapshot has rotated away still receives configuration and state together
|
||||
- Keep configuration bytes out of WAL entry payloads entirely — only the checkpoint file carries them
|
||||
- Observable: any checkpoint file, marker-triggered or periodic, contains a configuration-bytes section whose content hash matches the currently active configuration; two independent materializations at the same marker boundary and configuration produce byte-identical checkpoint files
|
||||
- _Requirements: 2.1, 2.3, 2.4, 5.1_
|
||||
|
||||
- [ ] 3. (P) Restore a host admin-state frame safely regardless of whether the host has ever joined this member
|
||||
- Look up the host by name in the live host table; if present, OR the restored admin-status bits into its current status
|
||||
- If absent, create a minimal placeholder entry (unjoined, dead-until-connected) carrying only the restored admin-status bits, using the same host-creation mechanism the existing synthetic sentinel host already uses without requiring a live join
|
||||
- Ensure the placeholder is valid enough that the host's real, later join is treated as an ordinary reconnect rather than a first-time join, so its existing admin-bit-preserving behavior applies unchanged
|
||||
- Guard against a retried restore attempt re-creating or conflicting with a placeholder an earlier, partially-failed attempt already created — find first, create only if truly missing
|
||||
- Observable: restoring a host admin-state frame for a host with no local record creates exactly one placeholder entry carrying the correct admin bits; restoring it again (simulating a retry) does not create a duplicate or error; when that host later joins for real, its admin status (e.g. closed) is still set afterward
|
||||
- _Requirements: 1.10, 1.11, 5.8_
|
||||
- _Boundary: Host Restore (worker.c)_
|
||||
|
||||
- [ ] 4. Decode host and queue admin-state frames on checkpoint load
|
||||
- Add host-frame decoding to the checkpoint load path, dispatching each decoded frame to the host restore capability
|
||||
- Update queue-frame decoding to read the new QueueAdminState encoding instead of the old raw layout, without changing the existing queue-state-restore behavior itself
|
||||
- Preserve the existing restore discipline: every frame in the checkpoint restores successfully before any position is advanced; any restore failure leaves prior state untouched
|
||||
- Observable: loading a checkpoint containing host and queue admin-state frames restores every host and queue's admin status correctly regardless of restore order between entity types
|
||||
- _Requirements: 1.10_
|
||||
- _Depends: 2.1, 2.2, 3_
|
||||
|
||||
- [ ] 5. Materialize the upgrade snapshot when a configuration-change marker is applied
|
||||
- Add explicit handling for the configuration-change marker in the single, unified entity-apply dispatch point (the same seam every member's apply already routes through — primary, standby, and restart replay alike), replacing its current no-op treatment
|
||||
- Trigger a forced checkpoint at that exact point, so materialization happens immediately and deterministically at the marker's boundary rather than waiting for a periodic trigger
|
||||
- Ensure the forced checkpoint shares the existing single-in-flight checkpoint guard, so a concurrent periodic checkpoint and this one can never both proceed at once
|
||||
- Treat a "checkpoint already in progress" outcome (e.g. a second configuration marker committing while an earlier one's checkpoint is still running) as an expected, self-resolving condition, not a failure to retry specially
|
||||
- Ensure this trigger only materializes state — it must not itself change how entities are interpreted or re-homed under the new configuration
|
||||
- Observable: applying a configuration-change marker produces a checkpoint at that marker's exact position, on both the primary and a standby, without requiring a controller restart; a second marker committing while the first's checkpoint is still running does not corrupt or lose either checkpoint
|
||||
- _Requirements: 1.1, 1.5, 1.8, 4.4, 5.2, 5.3_
|
||||
- _Depends: 2.1, 2.2, 2.3_
|
||||
|
||||
- [ ] 6. Core: activation fence gating serving and promotion on materialization
|
||||
- [ ] 6.1 (P) Add a materialization-and-hash-verification fence independent of the existing configuration-byte fence
|
||||
- Introduce a new fence condition distinct from the existing configuration-byte-mismatch fence, since "bytes don't match" and "materialization for a matching configuration isn't done yet" are different conditions that must be distinguishable
|
||||
- Verify by reading the current committed configuration-change marker and this member's own materialized-and-verified state for it, always re-reading the live, currently-active configuration bytes on disk on every verification attempt rather than trusting a previously-embedded or cached copy
|
||||
- Set the fence and log the actionable mismatch loudly on failure; clear it on success; never terminate the process on this condition
|
||||
- Observable: a member whose materialization for the current marker is incomplete or mismatched shows the fence set and a loud log entry naming the marker; once materialization completes and bytes match, the fence clears without any process restart
|
||||
- _Requirements: 1.8, 3.2, 3.4_
|
||||
- _Boundary: Activation Fence (ha.c/ha.h)_
|
||||
|
||||
- [ ] 6.2 Gate primary request-serving and promotion on the new fence
|
||||
- Extend the primary request-dispatch gate with this new fence check, composed alongside the existing configuration-byte fence rather than merged into it, admitting replication and read-only operations while fenced and refusing state-changing client requests
|
||||
- Extend the promotion gate the same way, so a member whose materialization is incomplete cannot become primary and immediately violate the serving gate the moment it wins an election
|
||||
- Introduce no new promotion-eligibility mechanism for WAL/log completeness — this fence is about materialization completeness only, layered on top of the existing Raft log-freshness election gate, not replacing or duplicating it
|
||||
- Wire a non-blocking, self-triggering retry so the fence re-verifies and clears automatically once conditions allow, without any operator action
|
||||
- Observable: a fenced primary continues answering read-only and status requests but rejects a state-changing client request with a not-ready response; a fenced member that wins an election does not begin serving under the new configuration until the fence clears on its own
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 4.3, 5.5_
|
||||
- _Depends: 6.1_
|
||||
|
||||
- [ ] 7. Remove the per-node restart/replay fallback and replace it with the marker-triggered materialization
|
||||
- Delete the configuration-upgrade handler's restart-request step now that materialization is driven entirely by the marker-apply trigger, decoupled from that request-handling code path
|
||||
- Add a deterministic test seam that can force a leadership change to land exactly between a configuration-change marker's commit and every member completing its snapshot install, for later regression use
|
||||
- Observable: pushing a configuration upgrade no longer restarts any controller process, and every member reaches the new configuration through the marker-apply trigger alone
|
||||
- _Requirements: 1.2, 1.3, 1.4, 2.2, 5.4, 6.3_
|
||||
- _Depends: 5_
|
||||
|
||||
- [ ] 8. Validation: regression coverage for every new mechanism
|
||||
- [ ] 8.1 (P) Unit-test the marker-apply trigger and its interaction with WAL retention
|
||||
- Assert the configuration-marker case fires a checkpoint exactly once per apply and is safe under a forced retry
|
||||
- Assert the trigger does not itself re-home entities
|
||||
- Add a deterministic fault-injection seam at the marker-apply-to-checkpoint-durable boundary, mirroring the existing fault-hook pattern already used elsewhere in this codebase, for later chaotic-scenario use
|
||||
- Assert a snapshot encoded in an unrecognized format version (simulating a rolling binary upgrade or the prior raw queue-frame layout) is rejected fail-loud, never partially installed
|
||||
- Assert that when a standby is genuinely mid-catch-up (has not yet acknowledged the marker's position) at the moment the marker-triggered checkpoint's WAL trim runs, that standby's still-needed WAL segments are not deleted
|
||||
- Observable: the new test suite passes, demonstrating exactly-once checkpoint firing, safe retry, format-mismatch rejection, and trim safety under the new trigger path
|
||||
- _Requirements: 1.1, 1.5, 1.8, 5.2, 5.3, 5.6, 5.7, 6.3, 7.3, 7.4_
|
||||
- _Depends: 5_
|
||||
- _Boundary: Marker Apply Trigger (test_upgrade_snapshot_apply.c)_
|
||||
|
||||
- [ ] 8.2 (P) Unit-test host placeholder restore and rejoin preservation
|
||||
- Assert restoring a host admin-state frame for a never-joined host creates a placeholder carrying the correct admin bits
|
||||
- Assert a retried restore after a simulated partial failure finds and reuses the existing placeholder rather than conflicting with it
|
||||
- Assert that a host's real first join after a placeholder was created preserves the admin status through the ordinary reconnect path
|
||||
- Observable: the new test suite passes, demonstrating create-if-missing correctness, idempotent retry, and admin-status survival through a real join
|
||||
- _Requirements: 1.10, 1.11, 5.8, 7.1_
|
||||
- _Depends: 3_
|
||||
- _Boundary: Host Restore (test_upgrade_snapshot_restore.c)_
|
||||
|
||||
- [ ] 8.3 (P) Unit-test the activation fence
|
||||
- Assert the fence sets on incomplete materialization or a configuration-hash mismatch and clears on successful verification
|
||||
- Assert the request-dispatch gate's allowlist behavior while fenced, and that promotion is refused while fenced
|
||||
- Observable: the new test suite passes, demonstrating fence set/clear transitions and correct gating of both serving and promotion
|
||||
- _Requirements: 3.2, 3.3, 3.4, 4.3, 5.5_
|
||||
- _Depends: 6.2_
|
||||
- _Boundary: Activation Fence (test_upgrade_activation_fence.c)_
|
||||
|
||||
- [ ] 8.4 (P) Build a cross-member active-configuration comparator and extend checkpoint verification tooling
|
||||
- Build a new offline, file-based tool that compares each member's active configuration content hash, recomputed fresh from each member's own on-disk configuration file, mirroring the existing checkpoint-comparison tool's shape
|
||||
- Extend the existing checkpoint-structure verification tool to decode and assert the embedded configuration reference's tag and hash (not merely report their presence), and to report host-frame count and configuration-bytes-section presence
|
||||
- Observable: the comparator correctly reports two members with matching configuration as identical and two with differing configuration as divergent; the extended verification tool reports a concrete tag/hash value and section counts for a real checkpoint file
|
||||
- _Requirements: 4.2_
|
||||
- _Depends: 2.3_
|
||||
- _Boundary: Regression Tooling (configcompare, snapverify)_
|
||||
|
||||
- [ ] 8.5 Reproduce bug-857 on the pre-change behavior and confirm it no longer occurs
|
||||
- Run the original materialized-state-divergence repro scenario against the pre-change code path and confirm the divergence occurs
|
||||
- Run the same scenario against the post-change behavior and confirm every member's materialized state converges identically
|
||||
- Observable: the same repro scenario demonstrably fails on the old behavior and passes on the new one
|
||||
- _Requirements: 6.2_
|
||||
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
|
||||
|
||||
- [ ] 8.6 Exercise a configuration upgrade interleaved with chaotic multi-kill failover
|
||||
- Drive a configuration upgrade while repeatedly killing and restarting the primary, and assert every member's materialized state and active configuration are identical afterward
|
||||
- Assert a host's administratively-set status specifically survives the same scenario, not merely inferred from the aggregate state check
|
||||
- Observable: the chaotic-upgrade soak run reports zero materialized-state divergence, zero configuration divergence, and correct host admin-status across all members
|
||||
- _Requirements: 4.1, 4.3, 6.1, 6.4_
|
||||
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
|
||||
|
||||
- [ ] 8.7 Inject a crash and a leadership change at the two critical upgrade windows
|
||||
- Force a controller crash/restart between a configuration-change marker's commit and that member's own snapshot write becoming durable, and assert materialization resumes correctly from the marker on restart
|
||||
- Force a leadership change between a configuration-change marker's commit and every member completing its snapshot install, and assert the new leader continues driving materialization to completion without any special resume logic
|
||||
- Observable: both fault-injection scenarios complete with the post-upgrade consistency invariant and the crash-safety invariant intact
|
||||
- _Requirements: 5.2, 5.4, 5.5, 6.3_
|
||||
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
|
||||
|
||||
- [ ] 8.8 Exercise a second configuration upgrade committing before the first's installation completes everywhere
|
||||
- Commit a second configuration-change marker while the first upgrade's snapshot is still installing on at least one member, and assert the earlier upgrade's in-flight state is cleanly superseded rather than left partially applied or conflated with the newer one
|
||||
- Observable: after both markers settle, every member reflects only the newer configuration and state, with no trace of a partially-applied intermediate state
|
||||
- _Requirements: 4.4, 7.2_
|
||||
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
|
||||
|
||||
- [ ] 8.9 Exercise the WAL-trim-under-marker-trigger scenario under chaotic conditions
|
||||
- Repeat the mid-catch-up WAL-trim assertion from the unit-level fixture, but under the full chaotic multi-kill soak rather than an isolated fixture
|
||||
- Observable: across a full chaotic-soak run with configuration upgrades in flight, no standby ever loses a WAL segment it still needed for catch-up
|
||||
- _Requirements: 7.4_
|
||||
- _Depends: 7, 8.1, 8.2, 8.3, 8.4_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- **Build environment (bug-918 recurrence 4, fixed in this worktree):** `nix develop` was hard-broken — `xbuild`'s `capnpc` package pins its `src` to the MUTABLE `main` ref of `shen390s/c-capnproto` with a hardcoded sha256, so it hash-mismatches every time that branch moves upstream (recurred 3x before in this same worktree per `.wolf/buglog.json`). Fixed durably by overriding `capnpc`'s `src` in `src/nixSupport/xce.nix` to a `pkgs.fetchFromGitHub` pinned at an immutable commit (`rev`+`sha256`) instead of tracking `main`. Any future task in this spec that needs a fresh `nix develop` should not hit this again; if it does, the fix pattern is the same (re-pin to the then-current upstream commit).
|
||||
- **Test-runner gotcha:** running `controller_unit_test` with default ASan settings hangs indefinitely in this sandbox — LeakSanitizer's stop-the-world thread scan never completes (confirmed: 50+ min wall clock, ~5s CPU, `futex_do_wait`). Always run with `ASAN_OPTIONS=detect_leaks=0` (matches the project's own meson test registration). Do not wrap the binary in `stdbuf` or anything else that adds its own `LD_PRELOAD` — conflicts with ASan's preload requirement and the binary refuses to start.
|
||||
- **After any `SNAPSHOT_FORMAT_VERSION` bump:** grep the whole `testsuite/` tree for other hardcoded copies of the old format-version literal before considering the bump done — `test_persist_snapshot_entities.c` (a different, already-shipped spec's test file) had its own independent `SNAPENT_TEST_FORMAT_VERSION` constant that silently went stale after task 1's bump and broke 3 unrelated tests until caught.
|
||||
|
|
@ -1628,8 +1628,17 @@ static int persist_write_checkpoint_marker(persist_t* p)
|
|||
* [uint32_t job_count]
|
||||
* [job_count * { uint32_t payload_len; uint8_t capnp_Job_bytes[payload_len] }]
|
||||
* [uint32_t queue_count]
|
||||
* [queue_count * { uint32_t name_len; char name[name_len]; uint32_t status }] */
|
||||
#define SNAPSHOT_FORMAT_VERSION 1u
|
||||
* [queue_count * { uint32_t name_len; char name[name_len]; uint32_t status }]
|
||||
*
|
||||
* config-upgrade-snapshot task 1 (Req 1.9, 5.7): bumped 1->2 to reserve the
|
||||
* version for the HostAdminState/QueueAdminState capnp migration -- the
|
||||
* queue-frame layout above (name_len/name/status raw bytes) is the PRIOR
|
||||
* layout this version now hard-rejects; the byte-layout migration itself
|
||||
* (new host section, queue frames re-encoded as capnp) lands in tasks
|
||||
* 2.1/2.2. A binary built against version 2 must never misinterpret a
|
||||
* checkpoint written under version 1's raw queue-frame layout (Design
|
||||
* Invariant 4: fail loud, never silently reinterpret). */
|
||||
#define SNAPSHOT_FORMAT_VERSION 2u
|
||||
|
||||
typedef struct {
|
||||
uint8_t* buf;
|
||||
|
|
|
|||
|
|
@ -3282,7 +3282,7 @@ static void test_snapshot_recv_two_chunks_assembles_and_renames(void)
|
|||
* header) but is now correctly rejected as an unrecognized
|
||||
* format_version. Layout (design.md "Checkpoint File Layout"): the
|
||||
* 57-byte legacy header (cfgref_present=0 at byte 16, so bytes 17-56 are
|
||||
* unread and left zero) + format_version=1 + job_count=0 + queue_count=0
|
||||
* unread and left zero) + format_version=2 + job_count=0 + queue_count=0
|
||||
* (69 bytes total), padded with zero filler out to 128 bytes to keep the
|
||||
* two-64-byte-chunk shape this test exercises — the padding is never
|
||||
* read (load_snapshot_entities stops right after the queue_count=0
|
||||
|
|
@ -3294,7 +3294,8 @@ static void test_snapshot_recv_two_chunks_assembles_and_renames(void)
|
|||
memcpy(body, "XCESNAP\0", 8);
|
||||
memcpy(body + 8, &snap_lsn, 8);
|
||||
/* body[16] (cfgref_present) left 0 by the memset above. */
|
||||
uint32_t fake_fmt = 1u; /* must match persist.c's SNAPSHOT_FORMAT_VERSION */
|
||||
uint32_t fake_fmt = 2u; /* must match persist.c's SNAPSHOT_FORMAT_VERSION
|
||||
* (config-upgrade-snapshot task 1 bumped 1->2) */
|
||||
uint32_t fake_job_count = 0u;
|
||||
uint32_t fake_queue_count = 0u;
|
||||
memcpy(body + 57, &fake_fmt, sizeof(fake_fmt));
|
||||
|
|
@ -3376,6 +3377,107 @@ static void test_snapshot_recv_two_chunks_assembles_and_renames(void)
|
|||
printf("test_snapshot_recv_two_chunks_assembles_and_renames: PASS\n");
|
||||
}
|
||||
|
||||
/* config-upgrade-snapshot task 1 (Req 1.9, 5.7): SNAPSHOT_FORMAT_VERSION was
|
||||
* bumped 1->2 to reserve the version for the HostAdminState/QueueAdminState
|
||||
* capnp migration of the host/queue admin-state frames. A checkpoint written
|
||||
* under the PRIOR format_version (1 -- the raw queue-frame layout this spec
|
||||
* migrates off of) must be hard-rejected fail-loud by a binary built against
|
||||
* the bumped SNAPSHOT_FORMAT_VERSION=2, never silently reinterpreted (Design
|
||||
* Invariant 4; persist.c:2541's existing `fmt != SNAPSHOT_FORMAT_VERSION`
|
||||
* check, unchanged logic, new constant). Same well-formed-except-for-
|
||||
* format-version fixture shape as
|
||||
* test_snapshot_recv_two_chunks_assembles_and_renames above, but the
|
||||
* embedded format_version is pinned to the literal PRIOR value (1)
|
||||
* regardless of whatever persist.c's SNAPSHOT_FORMAT_VERSION currently is —
|
||||
* this is exactly the "old on-disk checkpoint, new binary" skew the version
|
||||
* bump must reject. */
|
||||
static void test_snapshot_recv_old_format_version_rejected_fail_loud(void)
|
||||
{
|
||||
char tmpdir[] = "/tmp/xcectest_snaprecv_oldfmt_XXXXXX";
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmpdir));
|
||||
|
||||
char snap_dir[512];
|
||||
snprintf(snap_dir, sizeof(snap_dir), "%s/snap", tmpdir);
|
||||
TEST_ASSERT_EQUAL_INT(0, mkdir(snap_dir, 0700));
|
||||
|
||||
uint64_t snap_lsn = 200;
|
||||
size_t total_size = 128;
|
||||
uint8_t body[128];
|
||||
memset(body, 0, sizeof(body));
|
||||
memcpy(body, "XCESNAP\0", 8);
|
||||
memcpy(body + 8, &snap_lsn, 8);
|
||||
/* body[16] (cfgref_present) left 0 by the memset above. */
|
||||
uint32_t old_fmt = 1u; /* the pre-migration raw queue-frame layout --
|
||||
* deliberately NOT persist.c's current
|
||||
* SNAPSHOT_FORMAT_VERSION */
|
||||
uint32_t fake_job_count = 0u;
|
||||
uint32_t fake_queue_count = 0u;
|
||||
memcpy(body + 57, &old_fmt, sizeof(old_fmt));
|
||||
memcpy(body + 61, &fake_job_count, sizeof(fake_job_count));
|
||||
memcpy(body + 65, &fake_queue_count, sizeof(fake_queue_count));
|
||||
|
||||
uint8_t chunk0[64];
|
||||
uint8_t chunk1[64];
|
||||
memcpy(chunk0, body, 64);
|
||||
memcpy(chunk1, body + 64, 64);
|
||||
|
||||
persist_t *p = calloc(1, sizeof(persist_t));
|
||||
TEST_ASSERT_NOT_NULL(p);
|
||||
persist_config_t pcfg = persist_config_defaults();
|
||||
persist_set_workdir(p, tmpdir, "ctrl-snaprecv-oldfmt");
|
||||
persist_init(p, &pcfg, NULL);
|
||||
persist_start(p);
|
||||
|
||||
ha_state_t ha;
|
||||
snap_recv_ha_init(&ha);
|
||||
|
||||
/* Chunk 0 — assembling snap.part never itself parses the body, so this
|
||||
* must still succeed regardless of what format_version the bytes carry. */
|
||||
snapshot_transfer_t c0;
|
||||
memset(&c0, 0, sizeof(c0));
|
||||
c0.snapshot_lsn = snap_lsn;
|
||||
c0.chunk_index = 0;
|
||||
c0.chunk_offset = 0;
|
||||
c0.total_size = (uint64_t)total_size;
|
||||
c0.is_last = 0;
|
||||
|
||||
int rc = ha_handle_snapshot_chunk(&ha, p, snap_dir, &c0, chunk0, 64);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc, "assembling a chunk never parses the body");
|
||||
|
||||
/* Chunk 1 (isLast=true) triggers rename + persist_load_snapshot — this is
|
||||
* where the format-version check fires and must fail loud. */
|
||||
snapshot_transfer_t c1;
|
||||
memset(&c1, 0, sizeof(c1));
|
||||
c1.snapshot_lsn = snap_lsn;
|
||||
c1.chunk_index = 1;
|
||||
c1.chunk_offset = 64;
|
||||
c1.total_size = (uint64_t)total_size;
|
||||
c1.is_last = 1;
|
||||
|
||||
rc = ha_handle_snapshot_chunk(&ha, p, snap_dir, &c1, chunk1, 64);
|
||||
TEST_ASSERT_TRUE_MESSAGE(rc != 0,
|
||||
"a checkpoint encoded under the prior SNAPSHOT_FORMAT_VERSION must be "
|
||||
"rejected fail-loud by a binary built against the bumped version, "
|
||||
"never silently reinterpreted (Requirement 5.7)");
|
||||
|
||||
/* A rejected load must never advance applied_lsn (no partial/silent
|
||||
* acceptance of mismatched-format bytes). */
|
||||
uint64_t got_lsn = (uint64_t)atomic_load_explicit(&ha.applied_lsn,
|
||||
memory_order_acquire);
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, got_lsn,
|
||||
"a rejected format-mismatched checkpoint must never advance applied_lsn");
|
||||
|
||||
ha_fini(&ha);
|
||||
persist_stop(p);
|
||||
persist_fini(p);
|
||||
free(p);
|
||||
|
||||
char cmd2[512];
|
||||
snprintf(cmd2, sizeof(cmd2), "rm -rf %s", tmpdir);
|
||||
system(cmd2);
|
||||
printf("test_snapshot_recv_old_format_version_rejected_fail_loud: PASS\n");
|
||||
}
|
||||
|
||||
/* Test 6.1: a legacy (pre-ha-entity-snapshots) position-only snapshot —
|
||||
* exactly the 16-byte XCESNAP magic+lsn header, no entity payload — driven
|
||||
* through the real receive handler end-to-end. persist_load_snapshot's
|
||||
|
|
@ -3571,7 +3673,8 @@ static void test_snapshot_recv_128kb_two_chunks(void)
|
|||
memcpy(chunk0, "XCESNAP\0", 8);
|
||||
memcpy(chunk0 + 8, &snap_lsn, 8);
|
||||
/* chunk0[16] (cfgref_present) already 0 from calloc. */
|
||||
const uint32_t fake_fmt = 1u; /* must match persist.c's SNAPSHOT_FORMAT_VERSION */
|
||||
const uint32_t fake_fmt = 2u; /* must match persist.c's SNAPSHOT_FORMAT_VERSION
|
||||
* (config-upgrade-snapshot task 1 bumped 1->2) */
|
||||
memcpy(chunk0 + 57, &fake_fmt, sizeof(fake_fmt));
|
||||
|
||||
/* ── Chunk 0: not last ── */
|
||||
|
|
@ -3662,7 +3765,7 @@ static void test_snapshot_recv_out_of_order_chunks_assemble_correctly(void)
|
|||
/* 4 chunks of 64 bytes each = 256-byte body. Must be a WELL-FORMED
|
||||
* entity-bearing checkpoint (same reason as
|
||||
* test_snapshot_recv_two_chunks_assembles_and_renames above): legacy
|
||||
* 57-byte header (cfgref_present=0) + format_version=1 + job_count=0 +
|
||||
* 57-byte header (cfgref_present=0) + format_version=2 + job_count=0 +
|
||||
* queue_count=0, then filler out to 256 bytes. The filler is
|
||||
* position-dependent (distinct per source byte, not per chunk) so ANY
|
||||
* chunk landing at the wrong file offset is immediately detectable via
|
||||
|
|
@ -3676,7 +3779,8 @@ static void test_snapshot_recv_out_of_order_chunks_assemble_correctly(void)
|
|||
memcpy(body + 8, &snap_lsn, 8);
|
||||
for (int i = 16; i < 256; i++)
|
||||
body[i] = (uint8_t)(i * 3 + 7); /* position-dependent, non-repeating */
|
||||
uint32_t fake_fmt = 1u;
|
||||
uint32_t fake_fmt = 2u; /* must match persist.c's SNAPSHOT_FORMAT_VERSION
|
||||
* (config-upgrade-snapshot task 1 bumped 1->2) */
|
||||
uint32_t fake_job_count = 0u;
|
||||
uint32_t fake_queue_count = 0u;
|
||||
memcpy(body + 57, &fake_fmt, sizeof(fake_fmt));
|
||||
|
|
@ -4268,6 +4372,9 @@ void test_config_repl(void)
|
|||
RUN_TEST(test_snapshot_transfer_stops_and_propagates_send_failure);
|
||||
/* Task 3.5c: standby snapshot receive handler */
|
||||
RUN_TEST(test_snapshot_recv_two_chunks_assembles_and_renames);
|
||||
/* config-upgrade-snapshot task 1: prior SNAPSHOT_FORMAT_VERSION rejected
|
||||
* fail-loud by a binary built against the bumped version */
|
||||
RUN_TEST(test_snapshot_recv_old_format_version_rejected_fail_loud);
|
||||
/* Task 6.1: legacy position-only file consumed end-to-end via the
|
||||
* receive handler — entities_restored=false path */
|
||||
RUN_TEST(test_snapshot_recv_legacy_file_advances_without_entities);
|
||||
|
|
|
|||
|
|
@ -375,7 +375,8 @@ static void test_snapent_fixture_seeds_jobs_and_queues(void)
|
|||
/* ------------------------------------------------------------------ */
|
||||
|
||||
#define SNAPENT_LEGACY_HEADER_BYTES (16 + 1 + 8 + 32)
|
||||
#define SNAPENT_TEST_FORMAT_VERSION 1u
|
||||
#define SNAPENT_TEST_FORMAT_VERSION 2u /* must match persist.c's SNAPSHOT_FORMAT_VERSION
|
||||
* (config-upgrade-snapshot task 1 bumped 1->2) */
|
||||
#define SNAPENT_MAX_DECODE_PAYLOAD (16u * 1024 * 1024)
|
||||
|
||||
/* Decodes every job frame in the checkpoint at `path` into `out`
|
||||
|
|
|
|||
|
|
@ -443,6 +443,25 @@ typedef struct {
|
|||
uint32_t prevCrc;
|
||||
} capnp_repl_envelope_t;
|
||||
|
||||
/* ── Upgrade snapshot admin-state frames (config-upgrade-snapshot task 1,
|
||||
* Req 1.9/5.7) ───────────────────────────────────────────────────────────
|
||||
* Frame-only wire types for the checkpoint's host/queue admin-state
|
||||
* sections -- carry only the administratively-set (name, status) pair,
|
||||
* never the full xce_worker_t/xce_queue_t (design.md Design Invariant 5: a
|
||||
* snapshot frame must never inherit a live struct's dynamic fields just
|
||||
* because reusing it is convenient). Neither has a corresponding "live" C
|
||||
* struct, so each gets its own frame-only counterpart here, same as
|
||||
* capnp_snap_hdr_t/capnp_persist_cfg_t/capnp_repl_envelope_t above. */
|
||||
typedef struct {
|
||||
char* name;
|
||||
uint32_t status;
|
||||
} capnp_host_admin_state_t;
|
||||
|
||||
typedef struct {
|
||||
char* name;
|
||||
uint32_t status;
|
||||
} capnp_queue_admin_state_t;
|
||||
|
||||
/* ── Config replication Cap'n Proto mapped types (Phase 1) ──────────────
|
||||
* C-side counterparts for the capnp encode/decode shims for config sync.
|
||||
* Struct tags match the forward declarations near the top of this file. */
|
||||
|
|
|
|||
|
|
@ -752,6 +752,30 @@ struct SnapshotHeader $C.mapname("capnp_snap_hdr_t") {
|
|||
headerChecksum @8 :UInt32;
|
||||
}
|
||||
|
||||
# ── Upgrade snapshot admin-state frames (config-upgrade-snapshot task 1,
|
||||
# Req 1.9/5.7) ──────────────────────────────────────────────────────────
|
||||
# Independently-extensible Cap'n Proto frames for the on-disk checkpoint's
|
||||
# host/queue admin-state sections (SNAPSHOT_FORMAT_VERSION 2, persist.c).
|
||||
# New admin-state fields (e.g. a lock reason or tag) get new ordinals under
|
||||
# capnp's own schema-evolution rules -- no future SNAPSHOT_FORMAT_VERSION
|
||||
# bump needed per field addition. Deliberately NOT mapname'd onto
|
||||
# Worker/QueueConf (design.md Design Invariant 5): those live runtime
|
||||
# structs bundle static attrs and dynamic state together, which a snapshot
|
||||
# frame must never carry (Requirement 1.7 excludes dynamic fields). Each
|
||||
# maps instead onto its own frame-only C counterpart in typemap.h, matching
|
||||
# the existing capnp_snap_hdr_t/capnp_persist_cfg_t/capnp_repl_envelope_t
|
||||
# convention already established below for schema-only wire concepts with
|
||||
# no corresponding "live" C struct.
|
||||
struct HostAdminState $C.mapname("capnp_host_admin_state_t") {
|
||||
name @0: Text;
|
||||
status @1: UInt32;
|
||||
}
|
||||
|
||||
struct QueueAdminState $C.mapname("capnp_queue_admin_state_t") {
|
||||
name @0: Text;
|
||||
status @1: UInt32;
|
||||
}
|
||||
|
||||
# ── HA heartbeat ────────────────────────────────────────────────────
|
||||
struct HeartbeatMsg $C.mapname("heartbeat_msg_t") {
|
||||
senderPriority @0: UInt32 $C.mapname("sender_priority");
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue