myxce/.kiro/specs/config-upgrade-snapshot/requirements.md

Requirements Document

Introduction

XCE's controller state machine (jobs, queues, hosts) is interpreted against the active configuration. Today a configuration upgrade pushes the new configuration, restarts each controller, and lets each member re-derive its post-upgrade state by replaying its pre-upgrade WAL under the new configuration — with no snapshot pinning an identical materialized state across members. Because replay is configuration-dependent (the lost_and_found queue/host re-homes entities whose configuration entity vanished), members can diverge in materialized state after an upgrade (bug-857), amplified by any WAL tail divergence.

This feature makes a configuration upgrade establish a full, entity-bearing state snapshot at the configuration-change boundary, embed the new configuration in it, and install that same snapshot on every member — so post-upgrade state is identical by construction and no member re-derives it. It was originally Requirement 8 of ha-term-safety and is split out because it has a hard prerequisite (entity-bearing snapshots) that does not yet exist.

Boundary Context

• In scope: a full entity-bearing state snapshot (the upgrade snapshot) materialized at the configuration-change marker's (term, lsn); embedding the new configuration in that snapshot; every member ending up holding it, via local materialization (already applied the marker) or the out-of-band snapshot-install transport (behind or freshly joining); an activation gate on the serving primary tied to its own materialization-and-hash-verification completing, not to Raft election eligibility; the post-upgrade materialized-state + active-configuration consistency invariant; superseding ha-term-safety R7's interim config-push byte delivery for the upgrade case.
• Out of scope:
  • The configuration-change marker and consensus ordering of the change — owned by ha-term-safety (Requirement 7).
  • The entity-serialization machinery that produces an entity-bearing snapshot — owned by entity-bearing-snapshots; this spec consumes it, with one explicit exception: this spec adds the new host admin-state frame and migrates the existing queue admin-state frame to Cap'n Proto (see "Host and queue admin-state frames now extensible" below) — a deliberate, narrow modification of that spec's shipped frame encoding, not a re-opening of its entity-serialization design.
  • The Raft protocol (term/log/commit) — owned by the Raft TLA+ Conformance library series.
  • Controller voting-set / membership changes — ha-config-bootstrap.
  • Initial/bootstrap configuration replication (config tag zero → first config) — owned by ha-config-bootstrap, deliberately NOT unified with this spec's marker+snapshot mechanism (considered and rejected 2026-07-23). Bootstrap has no pre-existing entity state to protect (a genuinely blank node) and a chicken-and-egg constraint this spec's mechanism cannot resolve: a configuration-change marker is a committed log entry, but at cold-start there is no log or elected leader yet — election itself must work with zero config (ha-config-bootstrap Req 2.9/3.3). Forcing bootstrap through marker-commit + snapshot-materialize + hash-verify would also regress single-node self-elect's immediate-serve behavior (Req 2.9). This spec's mechanism activates only for a configuration change on an already-running cluster with live entity state.
• Adjacent expectations:
  • Dependency on entity-bearing snapshots — SATISFIED (2026-07-09) by ha-entity-snapshots (21/21 tasks, fully approved). That spec landed write_snapshot/load_snapshot (persist.c) as a direct Cap'n-Proto entity-encode/decode of job and queue records at checkpoint time, restoring before advancing position — not the COW xzarray_freeze/thaw mechanism this line originally anticipated (that machinery remains dormant; ha-entity-snapshots took a different, already-shipped implementation path). This spec no longer needs to wait; verify write_snapshot's actual on-disk entity format (persist.c, ha-entity-snapshots design.md) before writing this spec's own design so it builds on the real interface, not the COW one originally assumed here.
  • Anchors on the (term, lsn) configuration-change marker from ha-term-safety.
  • Reuses the existing snapshot_transfer_s snapshot-install transport (chunked, (term, lsn)-anchored, hash-verified, install-before-activate).
  • Reuses the existing checkpoint machinery (persist.c persist_checkpoint/write_snapshot/schedule_checkpoint) as the materialization mechanism rather than a bespoke path: it already runs off the raft apply path via delaymgr (never blocks a vtask handler — see ha-async-io.md), already guards a single in-flight checkpoint with a CAS (checkpoint_in_progress), and already pins checkpoint_lsn to last_applied_lsn. It already caches a committed-config reference (cfgref_present/cfgref_tag/cfgref_sha, ha-term-safety task 4.2) in the snapshot header — this spec extends that to a full embedded configuration payload, it does not replace the mechanism.
  • Gap identified 2026-07-21, scope resolved 2026-07-23 — host frames are missing from the entity-bearing payload; this spec owns adding them. write_snapshot as shipped by ha-entity-snapshots calls only collect_job_frames/collect_queue_frames (persist.c) — there is no collect_host_frames and no host section in the on-disk format. This spec's own tasks add it, not a follow-up on ha-entity-snapshots's owners. Confirmed content scope: record only admin-issued status changes (open/closed/locked/tagged, via the existing XCE_WORKER_ADMIN_MASK — worker.c:275), never the full xce_worker_t (host attrs and resource values are worker-reported and change dynamically; Requirement 1.7 already excludes them, and a full-struct encode would also break Requirement 5.1 determinism). The existing capnp_encode_worker/Worker capnp message (persist.c:82, used today for WAL_WORKER_CTRL replication) is NOT the right vehicle — it bundles static attrs, admin status, and dynamic state (resvals_to_sched, dead_since) together. Restore does NOT mirror restore_queue_state's update-only assumption — queues are always config-provisioned before restore runs (collect_queue_frames's own doc comment), but hosts are NOT: a worker_t is created only by a live join (join.c:100-102, init_worker), never from hosts.yaml config directly, and worker_ctrl (worker.c:655-658) itself hard -ENOENTs an admin command against a host that has never joined. A member receiving the upgrade snapshot via the out-of-band install path (Requirement 1.2's "behind" case) may specifically not have locally replayed that host's original join record yet — that is the whole reason it needs the snapshot instead of local replay. So restore_host_state must be create-if-missing: if key_find(c->workermgr, name) misses, create a minimal placeholder via makeWorker() (unjoined/XCE_WORKER_UNAVAIL until the physical worker actually connects and apply_join_state populates the rest) and set only the admin-status bits on it — mirroring the exact makeWorker()-without-join pattern the synthetic lost_and_found sentinel host already uses (config_repl.c:1114-1130), not inventing a new mechanism. Design must also confirm this restore path is idempotent under a retried partial failure (a second attempt's makeWorker call must find and update the placeholder a prior failed attempt already created, not treat it as a duplicate-create conflict — mirroring restore_job's own -EEXIST-is-success idempotent-replay handling). resources is dropped from scope entirely (see Introduction) — c->resmgr entries (xce_resource_t, controller.capnp:200) are pure config-derived definitions with no admin-mutable field (no open/close/lock action exists for a resource definition), and the dynamic per-load values (r15s/r1m) already live in the same w->host.resvals bucket Requirement 1.7 excludes — there is no distinct "resources" entity needing its own admin-state capture.
  • Host and queue admin-state frames now extensible (capnp), decided 2026-07-23. Neither frame should be a fixed-shape dead end for future admin-state fields (e.g. a lock reason or tag added later). The queue frame today is a hand-rolled raw layout — { uint32_t name_len; name; uint32_t status } (snap_queue_frame_t, persist.c:1759) — with zero schema-evolution story: load_snapshot hard-rejects any SNAPSHOT_FORMAT_VERSION mismatch (persist.c:2541, by deliberate fail-loud design, task 5.1), so every future field addition would otherwise force another all-nodes format bump. Cap'n Proto already solves exactly this for job frames (Job message, persist.c:60) — new fields get new ordinals; old encoded blobs keep decoding under capnp's own schema-evolution rules, no format bump needed per field. Decision: define HostAdminState and QueueAdminState Cap'n Proto structs ({ name @0: Text; status @1: UInt32; } each, independently extensible) and use them for both frame types, migrating the queue frame off its current raw layout in the same pass. This requires exactly one SNAPSHOT_FORMAT_VERSION bump (covering host+queue together, not two separate migrations later) — an old on-disk snapshot written under the prior raw queue-frame layout becomes unreadable by the new binary until that node's next checkpoint cycle regenerates one, consistent with the existing fail-loud version-mismatch behavior (not a new risk this decision introduces). See the Boundary Context "Out of scope" note above: this is the one explicit, deliberate exception to "this spec consumes, not modifies, ha-entity-snapshots's machinery."
  • bug-1418 — RESOLVED, no longer a blocker. The 2026-07-21 note here (and the matching spec.json.blocked_on entry) treated bug-1418 as a still-open, unowned liveness deadlock requiring a task-0 prerequisite fix. That was stale: .wolf/buglog.json already recorded it fixed on 2026-07-09 (resolved via upstream raft-apply-decoupling tasks 5.1/5.3/7.2 + bug-1421, not a promotion-gate code change), and it was independently re-verified fresh on 2026-07-23 with a 10-round --chaotic soak (result=PASS, zero committed divergence, zero committed job loss) against a freshly rebuilt controller. spec.json.blocked_on has been cleared. Requirement 6's chaotic-multi-kill soak signal is not at risk of this bug's noise. See project memory project_bug1418_prework_decision.md.

Design Invariants

Standing rules for design.md and implementation, distilled from this spec's own requirements discussion (2026-07-23) — not requirements themselves, but constraints any design/implementation must respect:

1. Classify every entity type as config-provisioned or join-provisioned before writing its restore path. Queues are config-provisioned (exist once config loads, independent of any external actor) — update-only restore is safe (restore_queue_state). Hosts are join-provisioned (a worker_t exists only after a live actor registers, join.c:100-102) — restore must be create-if-missing (Requirement 1.10). Assuming update-only for a join-provisioned entity is the exact mistake this spec's own design caught and corrected; any future entity type added to the snapshot must be classified explicitly, not defaulted to whichever pattern was written last.
2. Frame restore order (job/queue/host) must not matter. Re-homing (lost_and_found) is decided at materialization time and baked into the encoded bytes; restore never re-derives it. Each entity type's restore must be independent of the others having already run.
3. Every restore path must be idempotent under partial-failure retry. Established for jobs (restore_job's -EEXIST-is-success) and required for the new host placeholder path (Requirement 5.8): a retried restore after a partial failure must find-and-continue, never duplicate-or-conflict.
4. A format/version mismatch is always fail-loud, never a silent reinterpretation. Already this codebase's convention for corruption (persist.c:2541, task 5.1); now explicitly extended to rolling-binary-upgrade skew on snapshot-install (Requirement 5.6) and the queue-frame layout migration itself (Requirement 5.7). Do not add a compat-shim decode path later that silently reinterprets old bytes under a new schema.
5. Never reuse an existing wire message wholesale just because it is convenient. capnp_encode_worker/Worker (persist.c:82) was the readily-available option for host frames and would have been wrong — it bundles static attrs, admin status, and dynamic state together. A snapshot frame's field set must be justified field-by-field against Requirements 1.6/1.7 (admin-state only, dynamic excluded), never inherited by convenience from a message built for a different purpose (replication).
6. A placeholder entity created out-of-band (Requirement 1.10) must be safe for its eventual real counterpart to adopt. The lost_and_found sentinel host precedent (config_repl.c:1114-1130) and the host-frame placeholder both create a worker_t via makeWorker() outside the normal join flow; the later real join must route through the same admin-bit-preserving path (rejoin_worker) an ordinary reconnect uses (Requirement 1.11), and the placeholder must carry a valid, non-NULL hostattrs_t so that path's direct dereferences don't crash. Note this codebase already has two independent admin-bit-preserving mechanisms (wobj.c's override_worker and controller/worker.c's rejoin_worker, duplicating the same XCE_WORKER_ADMIN_MASK list) — integrate with the one join.c actually dispatches to, don't assume the other's semantics apply.

Requirements

Requirement 1: Full snapshot at the configuration-change boundary

Objective: As an operator, I want a configuration upgrade to materialize a full state snapshot at the change boundary, so that every member adopts the same post-upgrade state rather than re-deriving it.

Acceptance Criteria

1. When a member applies a committed configuration-change marker, the Controller shall materialize a full state snapshot (the "upgrade snapshot") at that marker's (term, lsn) boundary — apply, not commit, since checkpoint_lsn is already defined as the apply cursor (last_applied_lsn), and apply is decoupled from and can lag commit under load (raft-apply-decoupling).
2. The Controller shall ensure every member ends up holding the upgrade snapshot for the current marker — via that member's own local materialization once it has applied the marker itself, or via the out-of-band snapshot-install transport for a member that has not yet applied the marker (behind or freshly joining) — with both paths producing identical content per Requirement 5.1, never a mix of stale local state and a differently-sourced remote snapshot.
3. The Controller shall not require any member to re-derive post-upgrade state by replaying pre-upgrade WAL entries under the new configuration.
4. The Controller shall transfer the upgrade snapshot through the out-of-band snapshot-install path, not through WAL entry payloads.
5. The Controller shall trigger snapshot materialization synchronously upon applying the configuration-change marker (a forced checkpoint at that exact (term, lsn)), executed through the existing asynchronous checkpoint mechanism rather than inline within the raft apply path, so that the trigger is immediate and deterministic while the write itself never blocks a vtask handler.
6. The Controller shall include each host's administratively-set status (e.g., open/closed, locked, tagged) in the materialized snapshot, so that admin-set host state survives a configuration upgrade the same way admin-set queue status already does.
7. The Controller shall exclude dynamic, non-administratively-set host metrics (e.g., load averages, real-time resource utilization samples) from the materialized snapshot, since these naturally re-populate after restart and pinning them would make the snapshot's content non-deterministic (Requirement 5.1) for no correctness benefit.
8. Applying the configuration-change marker shall advance the Controller's apply position and trigger materialization (Requirement 1.5) but shall NOT itself change how the Controller interprets or re-homes entities (e.g., lost_and_found re-homing) under the new configuration — that interpretive switch is the activation step gated by Requirement 3.2, never an immediate side effect of apply. Without this separation, Requirement 3.3's "keep serving under the last-verified configuration" is incoherent: apply and activation must be distinct steps, not one.
9. The Controller shall encode the host and queue admin-state frames (Requirement 1.6) as independently-extensible Cap'n Proto structs rather than fixed raw byte layouts, so that a future admin-state field addition (e.g. a lock reason or tag) does not require another snapshot format-version migration.
10. If a host admin-state frame refers to a host with no existing entry in the local worker table at restore time (a member that has not yet locally replayed that host's original join record), the Controller shall create a minimal placeholder entry (unjoined, dead-until-connected) to carry the admin status, rather than failing the restore or discarding the host's admin state — mirroring the existing makeWorker()-without-join pattern already used for the synthetic lost_and_found sentinel host, never inventing a parallel mechanism.
11. When the physical worker for a placeholder host (Requirement 1.10) later joins for the first time, the Controller shall route that join through the same rejoin path (rejoin_worker) an already-known host uses — never the first-time-join path (init_worker) — so the placeholder's restored admin-status bits are preserved exactly as rejoin_worker already preserves them across an ordinary disconnect/reconnect (worker.c:374,390). The placeholder shall therefore be created with a valid, non-NULL hostattrs_t (even if minimal/zeroed): rejoin_worker's apply_join_state step directly dereferences the existing attrs struct with no null check (worker.c:376), and the underlying copy_worker allocation path (wobj.c:51-81) itself requires a non-NULL source attrs — omitting it crashes or -EINVALs on the very first real join, not merely on the placeholder-creation call.

Requirement 2: Configuration embedded in the upgrade snapshot

Objective: As an operator, I want the new configuration delivered together with the materialized state, so that a member never holds state and configuration from different points.

Acceptance Criteria

1. The Controller shall embed the new configuration in the upgrade snapshot so that installing the snapshot delivers configuration and materialized state as one consistent artifact.
2. The Controller shall make this upgrade snapshot supersede the interim out-of-band config-push byte delivery (ha-term-safety Requirement 7) for the upgrade case.
3. The Controller shall keep the configuration bytes out of WAL entry payloads (the WAL holds only the (term, lsn) configuration-change marker).
4. The Controller shall embed the currently active configuration in every checkpoint taken after a configuration change, not only the marker-triggered upgrade checkpoint, so that a member joining or catching up after the upgrade snapshot has been rotated away still receives configuration and materialized state together from whichever checkpoint it installs.

Requirement 3: Activation gating

Objective: As an operator, I want the serving primary — whichever member that is, self-promoted or freshly elected — to never serve client requests under a configuration whose materialized snapshot it has not itself completed and verified, so that Raft's log-completeness guarantee (who CAN be elected) is never mistaken for materialization-completeness (who SHOULD start serving).

Acceptance Criteria

1. The Controller shall rely on Raft's existing log-freshness election gate (a candidate's log must be at least as up-to-date as the voter's) to prevent a member missing the committed configuration-change marker from being elected; this spec introduces no separate promotion-eligibility mechanism for WAL/log completeness.
2. Regardless of whether a member authored the configuration-change marker itself or inherited it via replication and a later election, the Controller shall not let that member serve client requests under the new configuration until its own upgrade-snapshot materialization for the current marker has durably completed and the embedded configuration has been hash-verified against the marker.
3. While its own materialization for the current marker is incomplete, a member that is otherwise the serving primary shall continue serving under its own last-verified configuration rather than stopping service, consistent with the never-self-terminate / stay-alive invariant (controller/CLAUDE.md).
4. If the embedded configuration does not hash-verify against the marker, the Controller shall keep serving under the last-verified configuration (or remain in a loud non-serving fence if none exists yet) and continue retrying materialization rather than activating mismatched configuration.

Requirement 4: Post-upgrade consistency invariant

Objective: As a client and operator, I want every member identical after a configuration upgrade, so that an upgrade can never silently diverge materialized state across members.

Acceptance Criteria

1. When members complete a configuration upgrade, every member's materialized state shall be identical.
2. When members complete a configuration upgrade, every member's active configuration shall be identical.
3. While configuration upgrades interleave with failover, the Controller shall preserve the post-upgrade identical-state invariant across leadership changes.
4. If a second configuration-change marker commits before every member has completed installing the previous upgrade snapshot, the Controller shall have the new marker's materialization (Requirement 1.1) cleanly supersede the previous upgrade's in-flight distribution, never leave it partially in-flight or conflated with the new one.

Requirement 5: Crash-safety and idempotent resumption

Objective: As an operator, I want a configuration upgrade to survive a controller crash or leadership change at any point, so that correctness after failure is never traded away for upgrade progress.

Acceptance Criteria

1. The Controller shall make the upgrade snapshot's content a deterministic function of the configuration-change marker's (term, lsn) and the embedded configuration's content hash, so that independent materialization by any member — including a newly elected leader — yields identical content.
2. If a member crashes after the configuration-change marker commits but before that member's upgrade snapshot write completes durably, then upon restart the Controller shall resume materialization from the marker rather than serving under the new configuration with an incomplete or absent snapshot.
3. The Controller shall guard the forced upgrade checkpoint with the same single-in-flight-checkpoint idempotency mechanism used by periodic checkpoints, so that a post-crash retry or a concurrent periodic checkpoint cannot corrupt or partially overwrite the upgrade snapshot.
4. If leadership changes after the configuration-change marker commits but before every member has installed the upgrade snapshot, then the new leader shall resume driving materialization and distribution rather than treating the upgrade as complete or abandoning it.
5. Under no sequence of crashes, restarts, or leadership changes shall a member serve under the new configuration without having durably installed and hash-verified the matching upgrade snapshot (a failure-mode restatement of Requirement 3.2).
6. If the out-of-band snapshot-install transport would deliver a snapshot encoded in a format-version the receiving member's own binary does not recognize (e.g. a member not yet restarted onto the new binary during a rolling software upgrade), the Controller shall reject the install and fail loud rather than partially installing or misinterpreting the bytes, and shall retry once the receiving member is running a compatible binary/format — consistent with the never-self-terminate / stay-alive invariant (controller/CLAUDE.md).
7. After the host/queue admin-state frame migration to Cap'n Proto (Requirement 1.9), the Controller shall reject a snapshot encoded in the prior raw queue-frame layout with the same fail-loud, non-corrupting behavior as any other format-version mismatch, never silently reinterpreting old bytes under the new layout.
8. If a prior restore attempt already created a placeholder host entry (Requirement 1.10) before failing partway through the snapshot, a retried restore shall find and reuse that placeholder rather than treating its existence as a conflict — the same idempotent-replay guarantee restore_job's -EEXIST-is-success handling already provides for jobs.

Requirement 6: Reproduction and regression

Objective: As a maintainer, I want the bug-857 divergence reproduced and guarded, so that the fix is proven.

Acceptance Criteria

1. When the regression scenario interleaves a configuration upgrade with chaotic multi-kill failover, the test shall assert that every member's materialized state and active configuration are identical afterwards.
2. The test shall reproduce the bug-857 materialized-state divergence on the pre-fix code (config-push + per-node replay) and pass on the fixed code.
3. The test shall inject a controller crash/restart and a leadership change between configuration-change marker commit and upgrade-snapshot-install completion, and assert the post-upgrade consistency invariant (Requirement 4) and the crash-safety invariant (Requirement 5.5) still hold.
4. The regression test shall explicitly assert that a host's administratively-set status (open/closed/locked) survives the configuration-upgrade-plus-chaotic-failover scenario, not merely rely on the aggregate materialized-state equality check (Requirement 6.1) to imply it.

Requirement 7: Regression coverage for the host-placeholder, overlapping-upgrade, format-mismatch, and WAL-trim mechanisms

Objective: As a maintainer, I want the new mechanisms this spec introduces — host-placeholder creation and rejoin, overlapping upgrades, format-version mismatch handling, and the marker-triggered checkpoint's interaction with WAL-segment trim — exercised by their own regression tests, so they cannot silently regress the way bug-857 did before it had one.

Acceptance Criteria

1. The test suite shall exercise a host admin-status change (e.g. close) issued while that host is down, followed by a configuration upgrade whose snapshot restores on a member with no local record of that host's join, followed by the host's real first join, and assert the admin status is preserved through the placeholder-then-rejoin path (Requirements 1.10, 1.11).
2. The test suite shall exercise a second configuration-change marker committing before every member has finished installing the previous upgrade snapshot, and assert the previous upgrade's distribution is cleanly superseded rather than left partially in-flight or conflated (Requirement 4.4).
3. The test suite shall assert that a snapshot encoded in a format-version the receiving binary does not recognize — whether from a rolling binary upgrade (Requirement 5.6) or the prior raw queue-frame layout (Requirement 5.7) — is rejected fail-loud, never partially installed or silently reinterpreted.
4. The test suite shall exercise the marker-triggered forced checkpoint's WAL-segment trim (Requirement 1.1) while a standby is genuinely mid-catch-up (has not yet acked the marker's LSN), and assert that standby's still-needed WAL segments are not stranded — the existing retention-floor guard (repl_trim_lsn/trim_wal_segments, bug-1147) must hold identically under this new trigger path as it already does under the periodic WAL-threshold trigger, including under the chaotic multi-kill conditions Requirement 6 exercises.