lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/ha-config-bootstrap/requirements.md
Rongsong Shen 9bbff7b016 claude skills
2026-05-29 15:00:06 +08:00

15 KiB
Raw Permalink Blame History

Requirements: HA Config Bootstrap & Raft-Style Quorum Election

Introduction

The current LSF-style priority-based controller election has three critical safety gaps: (1) no guarantee of config freshness on the elected primary, (2) false self-elections during concurrent startup when the peer listener port is not yet open, and (3) no bootstrap path for a standby promoted during genuine failover whose local config is missing. This feature replaces priority-based election with Raft-style quorum voting, adds majority-commit WAL and config replication for full CP durability, establishes persistent standby-to-primary control channels, and hardens the controller startup sequence. The system shall maintain CP semantics: consistency over availability, with zero data loss on failover.

Boundary Context

  • In scope: quorum-based controller election, pre-election config bootstrap for unconfigured nodes, listener binding before election, startup pre-flight validation (peer count, certs, vcpus), majority-commit WAL replication with entry-type differentiated durability, majority-commit config push, persistent standby-to-primary control channels with catch-up synchronization, client-side transparent retry for transient controller errors, election term and voted_for durability across restarts, observable election and commit events via structured logs
  • Out of scope: Raft snapshot protocol (existing COW is orthogonal), joint consensus for cluster membership changes (candidate list is static at startup from --peers), mTLS between controllers (trusted-LAN model with server-side TLS is sufficient), config file format or storage layout changes
  • Adjacent expectations: the existing do_one_config_fetch() stub must be completed for config bootstrap to function in the deferred-fetch path; static controller peer membership (no dynamic join/leave) is a hard constraint; single-node mode (quorum=1) must remain fully operational without behavioral changes

Requirements

Requirement 1: Listener and Connection Framework Before Election

Objective: As a cluster operator, I want peer listener sockets to be open before election begins, so that concurrent-startup nodes can reach each other's vote handlers rather than falsely concluding peers are absent.

Acceptance Criteria

  1. When a controller starts, the system shall bind and listen on its peer port before initiating any election probe.
  2. When a controller starts, the system shall initialize the I/O manager and virtual task manager before initiating any election probe, so that incoming vote requests are handled by the normal dispatch loop.
  3. When the controller is in the startup-initializing or electing state, the system shall reject non-vote, non-config-probe requests with a not-ready error response.
  4. When the controller is in standby state, the system shall reject primary-only operations with a standby error response that includes a redirect to the known primary host and port.

Requirement 2: Quorum-Based Controller Election

Objective: As a cluster operator, I want the controller election to require a strict majority of peer votes, so that a minority partition cannot elect a primary and split-brain is prevented.

Acceptance Criteria

  1. When a controller enters the electing state, the system shall send vote requests to all peer controllers and require acknowledgments from a strict majority (greater than or equal to half the cluster size rounded up) before becoming primary.
  2. When a controller receives a vote request from a candidate with a lower term, the system shall deny the vote.
  3. When a controller receives a vote request from a candidate with a higher term, the system shall update its local term and clear any prior vote record for the previous term.
  4. When a controller receives a vote request from a candidate whose config tag or WAL log sequence number is less fresh than its own, the system shall deny the vote.
  5. When a peer controller is in catch-up synchronization and has not yet caught up to the primary's committed state, the system shall deny its vote in any election.
  6. When a controller has already granted a vote to one candidate in a given term, the system shall deny any subsequent vote request from a different candidate in the same term.
  7. When a candidate wins the vote count but any responding peer has a higher config tag or equal config tag with a higher WAL log sequence number, the system shall yield to that fresher peer and enter standby state.
  8. When a candidate with no local config (config tag zero) determines that any peer has a config with a non-zero tag, the system shall refuse to become primary and exit with a fatal error.
  9. When the cluster is a single node, the system shall self-elect as primary immediately with no quorum requirement.
  10. When a split vote occurs where no candidate achieves a quorum, the system shall retry the election once with a randomized delay and incremented term.
  11. If the second election attempt also fails to achieve a quorum, the system shall enter standby state.
  12. When every responding peer has an identical config tag and WAL log sequence number as the candidate, the system shall use randomized pre-election jitter as the tiebreaker without considering priority values.

Requirement 3: Config Bootstrap for Unconfigured Nodes

Objective: As a cluster operator, I want a controller node that starts with no local configuration to automatically discover and fetch configuration from the proven election winner after the election resolves, so that fresh nodes can join an existing cluster without manual intervention.

Acceptance Criteria

  1. When a controller starts with no local config (config tag zero), the system shall probe all peer controllers for their config tag before declaring candidacy.
  2. When a controller with no local config detects that at least one peer has a non-zero config tag, the system shall participate in the election as a voter only and defer the config fetch until the election winner is known.
  3. When a controller with no local config and all reachable peers also have no config, the system shall proceed to election as a normal candidate with config tag zero.
  4. When a controller with no local config cannot reach any peer, the system shall retry with exponential backoff (starting at 500 milliseconds, doubling up to 10 seconds) until a peer responds or the process is killed by an administrator.
  5. When the election resolves and a deferred config fetch is pending, the system shall fetch the full configuration from the proven election winner.

Requirement 4: Startup Validation and Safety Checks

Objective: As a cluster operator, I want the controller to validate its startup prerequisites before initialization, so that misconfigured deployments are detected immediately with clear, actionable error messages.

Acceptance Criteria

  1. When the --peers list contains an even number greater than 2, the system shall reject the configuration at startup with an error message explaining that an odd number of controllers is required for majority formation.
  2. When the TLS certificate or key file is missing or unreadable at the expected path, the system shall reject startup with an error message that includes the full file path.
  3. When the --vcpus value is less than 1, the system shall reject startup with an error message.
  4. When the cluster config file contains a vcpus key, the system shall log a deprecation warning directing operators to use the --vcpus CLI parameter instead.

Requirement 5: Majority-Commit WAL Replication

Objective: As a cluster operator, I want WAL entries for critical state changes to be committed only after a majority of controllers have durably written them, so that zero data loss is guaranteed on failover.

Acceptance Criteria

  1. When the primary writes a job submission, job state transition, or queue change WAL entry, the system shall not acknowledge the write to the caller until a majority of controllers (including the primary itself) have confirmed durable write of that entry.
  2. When the primary writes a host state or checkpoint reference WAL entry, the system shall acknowledge the write without waiting for standby confirmation.
  3. When a standby controller receives a WAL entry from the primary, the system shall write it to durable storage and reply with an acknowledgment carrying the written log sequence number.
  4. When a standby controller falls more than the configurable maximum pending acknowledgments behind the primary, the system shall remove that standby from the active replication set and trigger catch-up reconnection.
  5. If the primary cannot obtain majority acknowledgment within the configured commit timeout, the system shall return a timeout error to the caller.
  6. When a standby controller is promoted to primary, the system shall truncate any WAL entries above the primary's last known committed log sequence number before replaying the WAL.
  7. When a standby controller reconnects to a primary via catch-up, the system shall truncate any uncommitted WAL entries above the primary's committed log sequence number before requesting WAL replay.
  8. When the cluster is a single node, the system shall consider the local write alone as satisfying the majority requirement.

Requirement 6: Persistent Standby-to-Primary Control Channel

Objective: As a cluster operator, I want standby controllers to maintain a persistent connection to the primary, so that failover detection is sub-second and WAL replication avoids per-entry connection overhead.

Acceptance Criteria

  1. When a standby controller enters standby state after election, the system shall establish and maintain a persistent TLS connection to the elected primary.
  2. When the persistent control channel disconnects, the system shall detect the failure immediately and trigger a new election.
  3. When heartbeat messages are received on the persistent control channel, the system shall reset the heartbeat timeout and defer election.
  4. When no heartbeat is received within the election timeout plus randomized jitter on the persistent control channel, the system shall trigger a new election.
  5. When a standby with an outdated config tag or applied log sequence number connects to the primary, the system shall synchronize all missing state (config snapshot and WAL entries) before admitting the standby to the active replication set.
  6. When a standby's applied log sequence number is older than the primary's oldest retained WAL segment, the system shall transfer a full state snapshot before streaming WAL entries.
  7. When a standby connects with a config tag that differs from the primary's, the system shall push the primary's current configuration after catch-up synchronization completes.
  8. When a standby wins an election and becomes primary, the system shall close its outbound connection to the old primary and begin accepting inbound control channel connections from other standbys.

Requirement 7: Client-Side Transparent Retry for Transient Controller Errors

Objective: As a daemon developer, I want libxce to transparently retry requests when the controller returns transient errors, so that worker, scheduler, and API server code does not need ad-hoc retry loops.

Acceptance Criteria

  1. When libxce receives a not-ready error from a controller, the system shall retry the request with exponential backoff against the same endpoint.
  2. When libxce receives a standby error that includes a redirect to the current primary, the system shall reconnect to the redirected host and port and retry the request.
  3. When libxce receives a TCP-level connection refused or timeout error, the system shall reconnect and retry the request.
  4. When libxce receives a non-retriable error (authentication failure, invalid request, permission denied, not found), the system shall surface the error to the caller immediately without retry.
  5. The system shall provide configurable retry parameters (maximum retries, initial delay, maximum delay, backoff multiplier) with safe production defaults.
  6. When multiple clients retry simultaneously after a failover, the system shall stagger their retry attempts through exponential backoff to prevent thundering herd.

Requirement 8: Majority-Commit Config Replication

Objective: As a cluster operator, I want configuration changes pushed to the cluster to be committed only after a majority of controllers have durably written them, so that operator changes are not silently lost on primary failover.

Acceptance Criteria

  1. When the primary pushes a configuration upgrade, the system shall not acknowledge the change as committed until a majority of controllers (including the primary itself) have confirmed durable write.
  2. When a standby controller receives a configuration upgrade from the primary, the system shall write the config blob to durable storage and reply with an acknowledgment.
  3. If the primary cannot obtain majority acknowledgment for a configuration upgrade within the configured commit timeout, the system shall return a timeout error to the caller.
  4. When a standby is promoted to primary, the system shall serve only configurations up to the last known committed config tag received via heartbeat from the previous primary.
  5. When the cluster is a single node, the system shall consider the local write alone as satisfying the majority requirement for config upgrades.

Requirement 9: Election State Durability Across Restarts

Objective: As a cluster operator, I want the controller's election term and vote record to survive process restarts, so that a restarting node cannot double-vote in a term it already participated in.

Acceptance Criteria

  1. When a controller's term is incremented or its vote is granted to a candidate, the system shall durably persist the current term and voted-for candidate name before responding to the vote request.
  2. When a controller starts up, the system shall restore the most recently persisted term and voted-for record from durable storage before participating in any election.
  3. When a persisted term file does not exist at startup (first boot), the system shall initialize the term to zero with no prior vote recorded.

Requirement 10: Election and Replication Observability

Objective: As a cluster operator, I want election results, vote tallies, config bootstrap operations, WAL commit quorum status, and config push quorum status to appear in structured logs at INFO or WARN severity, so that cluster health and HA behavior can be monitored without external tooling.

Acceptance Criteria

  1. When an election completes, the system shall log at INFO level the outcome (primary won or standby), the term, the number of votes granted, and the quorum size.
  2. When a config bootstrap probe determines the config situation (all peers tag zero, config exists on peers, or no peer reachable), the system shall log the finding at INFO or WARN level.
  3. When a WAL commit achieves quorum, the system shall log at INFO level; when a WAL commit times out, the system shall log at WARN level.
  4. When a configuration upgrade achieves quorum, the system shall log at INFO level; when a configuration upgrade times out, the system shall log at WARN level.
  5. When a controller transitions between the init, electing, primary, and standby states, the system shall log the transition at INFO level.