update documents
This commit is contained in:
parent
c705cadadc
commit
d70bf6a402
51 changed files with 995 additions and 1997 deletions
|
|
@ -142,7 +142,8 @@ The full design is chosen for clusters where config integrity verification, chan
|
|||
auditability, and the push mechanism for online config updates are operational requirements.
|
||||
|
||||
**WAL-based config replication (considered and rejected)**: XCE already has a WAL
|
||||
replication path (`src/controller/repl.c`, ~340 lines) that streams job/queue/resource
|
||||
replication path (`src/controller/repl.c`, 881 lines as of this audit — grown from the
|
||||
~340 lines at design time) that streams job/queue/resource
|
||||
mutations from primary to standbys. Extending it to carry serialised `ClusterConfig`
|
||||
blobs would reuse the existing replication infrastructure. This approach is rejected
|
||||
for three reasons: (1) **Semantic mismatch** — WAL replication is continuous and
|
||||
|
|
@ -187,8 +188,12 @@ and `validate_host()` are complete; see `FUTURE_WORK.md`):
|
|||
- ~170 LOC
|
||||
|
||||
The `OP_upgradeConfig` handler is gated behind a compile-time flag
|
||||
`CONF_ENABLE_UPGRADE_CONFIG` (default: 0, disabled — see §5.3 safety gate). The
|
||||
gate MUST NOT be set to 1 in any production build until BOTH prerequisites are met:
|
||||
`CONF_ENABLE_UPGRADE_CONFIG`. **As designed**, the default is 0 (disabled — see
|
||||
§5.3 safety gate) and the gate MUST NOT be set to 1 in any production build
|
||||
until BOTH prerequisites are met. **As shipped**, `src/controller/config_repl.h`
|
||||
hardcodes the flag to `1` (enabled) with no Meson build option to override it —
|
||||
see the drift note in §5.3. Both prerequisites below remain unmet in the
|
||||
current codebase:
|
||||
|
||||
1. **`validate_host()`** is implemented with a real authorisation check (not a stub
|
||||
that returns success for all peers). Minimum viable check: verify the peer's
|
||||
|
|
@ -206,6 +211,11 @@ active, Phase 2 infrastructure compiled but inert). The gate is removed from the
|
|||
codebase when both prerequisites are merged and integration-tested. Phase 1 alone
|
||||
eliminates manual provisioning and config drift — the core problems stated in §1.
|
||||
|
||||
**This commitment is not currently met**: the codebase already ships with
|
||||
`CONF_ENABLE_UPGRADE_CONFIG` hardcoded to `1` while `validate_host()` remains
|
||||
the unconditional-success stub — the exact unsafe combination this gate was
|
||||
designed to prevent. See the drift note in §5.3.
|
||||
|
||||
---
|
||||
|
||||
## 3. Config Storage Layout
|
||||
|
|
@ -221,7 +231,7 @@ would require a separate atomic-rename technique anyway.
|
|||
|
||||
**Local filesystem only — NFS is explicitly unsupported.** The controller's working
|
||||
directory (including `conf/`, `data/wal/`, and `data/snap/`) MUST reside on a local
|
||||
filesystem. `persist_init()` (persist.c:217-222) detects and rejects NFS mounts at
|
||||
filesystem. `persist_init()` (persist.c:839-843) detects and rejects NFS mounts at
|
||||
startup with a FATAL error. Config replication assumes local `rename(2)` atomicity
|
||||
and `fsync(2)` durability guarantees; neither holds on NFS. No effort is made to
|
||||
support network or distributed filesystems for config storage. Cross-filesystem
|
||||
|
|
@ -615,6 +625,18 @@ half-implemented admin endpoint from being reachable in production before its
|
|||
authorisation check is complete. The compile-time gate is removed once
|
||||
`validate_host()` has a production implementation (tracked in `FUTURE_WORK.md`).
|
||||
|
||||
**Drift from shipped code**: The compile-time default described above was the
|
||||
design intent and is not what the current codebase does. As implemented,
|
||||
`src/controller/config_repl.h` defines `CONF_ENABLE_UPGRADE_CONFIG` to `1`
|
||||
unconditionally (comment: "enabled: standbys apply pushed config"), and
|
||||
`src/controller/meson.build` has no `-Denable-upgrade-config` option or any
|
||||
override of the flag — the compile-time default cannot be reverted to 0
|
||||
without editing the header. Meanwhile `validate_host()` (`src/controller/join.c`)
|
||||
is still the literal stub `return 1;` for every peer. `OP_upgradeConfig` is
|
||||
therefore reachable in any build of the current codebase without the
|
||||
authorisation check this gate was meant to require. This should be treated as
|
||||
an open safety gap, not a documentation nit.
|
||||
|
||||
### 5.4 Primary handler
|
||||
|
||||
The primary handles `OP_replConfig` via a new case in `cluster.c:myreq_handler()` (the
|
||||
|
|
@ -1320,6 +1342,17 @@ load_primary_config() INFO("Starting as standby
|
|||
cluster_start()
|
||||
```
|
||||
|
||||
> **Implementation note — startup ordering changed**: The diagram above
|
||||
> reflects this design's original proposal (election before
|
||||
> `iomgr_init`/`endpoint_init`). The shipped implementation moves the listener
|
||||
> and `iomgr_init`/`vtaskmgr_init` **before** election instead, closing a
|
||||
> 2–3 second window during which the port was unreachable to election probes.
|
||||
> See `.kiro/specs/ha-config-bootstrap/design.md`
|
||||
> (`Net[iomgr_init + cluster_open_listener + vtaskmgr_init]` runs before
|
||||
> `Quorum[am_i_primary_quorum]`) — this ordering change accompanies the
|
||||
> replacement of `am_i_primary()` with Raft-style quorum election (§9
|
||||
> archival note).
|
||||
|
||||
### 7.1 Primary config load — fail-fast behaviour
|
||||
|
||||
`load_primary_config()` replaces `get_default_conf()` on the primary path:
|
||||
|
|
@ -1385,6 +1418,10 @@ From the current `cluster_init()` (`cluster.c`):
|
|||
|
||||
### 8.2 What stays in phase 2 (after config)
|
||||
|
||||
> **Implementation note**: As shipped, `iomgr_init`/`endpoint_init` (and
|
||||
> `vtaskmgr_init`) run **before** election, not after — see the ordering note
|
||||
> in §7. The table below reflects this design's original phase-2 placement.
|
||||
|
||||
| Line(s) | Code | Phase |
|
||||
|---|---|---|
|
||||
| new (before iomgr) | Stop heartbeat listener: set `atomic_store(&c->heartbeat_listener_running, 0)` and `pthread_join(c->heartbeat_listener_tid, NULL)` — the join must complete before `iomgr_init` to free the listening port and prevent `EADDRINUSE` | 2 |
|
||||
|
|
@ -1460,288 +1497,26 @@ the primary's startup log and tune `max_attempts` × `max_delay_ms` accordingly.
|
|||
|
||||
## 9. am_i_primary() Implementation
|
||||
|
||||
The current `am_i_primary()` stub (`ha.c:213`) returns `c->ha.my_priority == 0`,
|
||||
which is never correct for a multi-node cluster — it never probes whether a
|
||||
higher-priority candidate is actually alive.
|
||||
|
||||
### 9.0 Simultaneous-boot symmetry breaking
|
||||
|
||||
When all candidates start simultaneously, nobody is primary yet and nobody responds
|
||||
to election probes. Without symmetry breaking, every candidate would expire its
|
||||
deadline and self-declare primary — a multi-way split-brain.
|
||||
|
||||
The mechanism has two parts:
|
||||
|
||||
**1. Dedicated heartbeat listener (all self-declared primaries)**
|
||||
|
||||
After Phase 1, **every candidate that `am_i_primary()` returns true for** launches a
|
||||
minimal heartbeat listener thread before entering Phase 2. This is not limited to
|
||||
priority-0 — if a lower-priority candidate wins the jitter race and self-declares
|
||||
primary, its heartbeat listener is essential for the higher-priority true primary
|
||||
to detect the split-brain during its own election probe. The listener is started
|
||||
in a dedicated pthread immediately after `am_i_primary()` returns:
|
||||
|
||||
```c
|
||||
if (am_i_primary()) {
|
||||
/* Start a minimal SSL listener that responds to OP_heartbeat probes.
|
||||
* This lets lower-priority candidates detect us while Phase 2 initializes. */
|
||||
start_heartbeat_listener(c); /* dedicated pthread — non-blocking poll loop */
|
||||
load_primary_config(c); /* FATAL if missing */
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
The heartbeat listener is fully specified in the parent HA design
|
||||
(`persistence-and-ha-design.md` Part 5). This section only describes the
|
||||
integration point relevant to config replication: the listener must be active
|
||||
before the standby's `fetch_config_from_primary()` connects, so the standby can
|
||||
reach a primary that is still in Phase 2 initialisation. Summary of the
|
||||
listener's properties (see parent design for full specification):
|
||||
|
||||
- Binds the port from `--peers` using `c->ha.ssl_ctx`
|
||||
- Non-blocking poll loop; accepts `OP_heartbeat` only; rejects other opcodes
|
||||
- Does NOT depend on iomgr, vtaskmgr, persist, or any Phase 2 component
|
||||
- Exits when the main thread clears `atomic_int heartbeat_listener_running`
|
||||
(set to 0 before `iomgr_init` in Phase 2). The main thread calls
|
||||
`pthread_join(heartbeat_listener_tid, NULL)` before `endpoint_init()` to
|
||||
prevent EADDRINUSE.
|
||||
- Registered in `cluster_t` (`c->heartbeat_listener_tid`,
|
||||
`c->heartbeat_listener_running`) so `cluster_fini()` can join the thread
|
||||
on any FATAL exit path before Phase 2
|
||||
|
||||
**2. Priority-based base delay + randomized jitter (all candidates)**
|
||||
|
||||
Each candidate's election timeout has two components: a deterministic
|
||||
**priority-based base delay** (higher priority = shorter delay) plus a small
|
||||
random jitter for tie-breaking:
|
||||
|
||||
```c
|
||||
/* Priority-based base delay: priority-0 waits 0ms, priority-1 waits
|
||||
* election_priority_step_ms, priority-2 waits 2× that, etc.
|
||||
* This ensures the highest-priority node nearly always wins without probing. */
|
||||
long to_ms = c->ha.config.election_timeout_ms;
|
||||
to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms;
|
||||
|
||||
/* Small random jitter for tie-breaking between same-step candidates */
|
||||
long jitter_ms = (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms);
|
||||
to_ms += jitter_ms;
|
||||
```
|
||||
|
||||
With `election_timeout_ms = 2000`, `election_priority_step_ms = 500`, and
|
||||
`election_jitter_max_ms = 200`, the effective deadlines are:
|
||||
- Priority-0: 2000ms–2199ms (self-declares primary first)
|
||||
- Priority-1: 2500ms–2699ms (probes priority-0 first; if alive, becomes standby)
|
||||
- Priority-2: 3000ms–3199ms (probes priority-0 and priority-1 first)
|
||||
|
||||
The priority-based base delay all but eliminates simultaneous split-brain during
|
||||
normal boots: priority-0 declares primary and starts its heartbeat listener
|
||||
~500ms before priority-1's deadline, so priority-1's probe finds a live primary.
|
||||
The priority-based step and jitter together make simultaneous split-brain
|
||||
vanishingly unlikely for N=3 in systemd-managed deployments:
|
||||
|
||||
- Priority-0 fires at t=0–199ms (election_timeout_ms + jitter only)
|
||||
- Priority-1 fires at t=500–699ms (step + jitter, probes priority-0 first)
|
||||
- Priority-2 fires at t=1000–1199ms (2×step + jitter)
|
||||
|
||||
Adjacent-priority candidates collide only if the lower-priority node starts so
|
||||
late that its jitter-extended deadline precedes the higher-priority node's first
|
||||
heartbeat response, which requires the 500ms step to be less than 2×J_max
|
||||
(400ms) — false for the configured values. For three nodes to all self-declare
|
||||
primary, all three must start within the same ~200ms jitter window, which is
|
||||
bounded by OS scheduler precision and systemd startup ordering.
|
||||
|
||||
The `election_priority_step_ms` is added to `ha_config_t` alongside the existing
|
||||
`election_timeout_ms` and `election_jitter_max_ms`. The jitter-only fallback
|
||||
(priority-0 with no step) retains the original behaviour when `election_priority_step_ms = 0`.
|
||||
|
||||
The window is bounded: in the worst case (all jitter values 0ms apart), all
|
||||
candidates declare primary. Once both reach Phase 2, the HA watchdog resolves
|
||||
the split via priority comparison in heartbeat exchange (see parent HA design).
|
||||
This is bounded by the time for the slower candidate to complete Phase 2
|
||||
(~4–13s per §8.5) plus one heartbeat interval (~500ms).
|
||||
|
||||
**Split-brain window during simultaneous boot**: The heartbeat listener starts
|
||||
immediately after `am_i_primary()` returns, but it is a minimal pre-iomgr
|
||||
listener that only handles `OP_heartbeat`. **All candidates that self-declare
|
||||
primary launch a heartbeat listener**, not just priority-0 — this is critical:
|
||||
if a lower-priority candidate wins the jitter race and self-declares primary,
|
||||
its heartbeat listener provides the response that resolves the split-brain
|
||||
when the higher-priority (true) primary probes it.
|
||||
|
||||
During simultaneous boot, all candidates start with `ha.term = 0`
|
||||
(`atomic_store_explicit(&ha->term, 0ULL, ...)` in `ha_init()`, ha.c:34).
|
||||
Term-based priority comparison cannot break symmetry at this stage — symmetry
|
||||
breaking during simultaneous boot relies entirely on the jitter mechanism
|
||||
(part 2 below). Term-based step-down becomes effective only after the first
|
||||
successful election, when the winning primary increments its term. The jitter
|
||||
window is therefore the primary defence against multi-way split-brain during
|
||||
cold start; the heartbeat listener provides the detection channel but term
|
||||
comparison is not yet discriminating.
|
||||
|
||||
If a heartbeat listener fails to bind its port (`EADDRINUSE`), the candidate
|
||||
logs a WARN indicating a likely split-brain condition (another candidate has
|
||||
already bound the port) and continues initialization. This is expected during
|
||||
simultaneous boot with overlapping jitter windows — the candidate that bound
|
||||
the port first keeps it; the other continues to Phase 2 where the HA watchdog's
|
||||
heartbeat exchange will resolve the split-brain.
|
||||
|
||||
If a lower-priority candidate also
|
||||
self-declares primary and enters Phase 2, the HA watchdog heartbeat exchange
|
||||
will resolve the split-brain once both candidates reach Phase 2 — but this
|
||||
requires the HA watchdog to compare candidate priority (from `--peers` ordering)
|
||||
in addition to term, since both nodes have term=0 during simultaneous boot.
|
||||
**This design implements** `ha_check_heartbeat_nocluster()` (ha.c:101-126)
|
||||
with a priority comparison at term=0 so that a lower-priority self-declared primary
|
||||
steps down when it receives a heartbeat from a higher-priority node with the
|
||||
same term. Without this enhancement, the current elapsed-time-only check cannot
|
||||
distinguish a legitimate primary from a split-brain peer during cold start.
|
||||
|
||||
**CRITICAL DEPENDENCY — implemented by this design**: The `ha_check_heartbeat_nocluster()`
|
||||
enhancement (priority comparison at term=0) is a **hard prerequisite** for correct
|
||||
simultaneous-boot behavior. Without it, a lower-priority self-declared primary will
|
||||
not step down when it receives a heartbeat from a higher-priority peer with the same
|
||||
term — the split-brain is unbounded, not 4–13s. This enhancement is implemented in
|
||||
this design (`ha.c` — see §12 files changed) alongside the election probing. The
|
||||
parent HA design (`persistence-and-ha-design.md` Part 5) specifies the heartbeat
|
||||
exchange requirements; this design provides the priority-comparison implementation
|
||||
which the parent design references. Test coverage:
|
||||
`test_am_i_primary_simultaneous_boot` in §14.1 validates priority-based step-down
|
||||
at term=0 during simultaneous boot.
|
||||
|
||||
**Implementation scope**: This design adds a `sender_priority` field to the
|
||||
`OP_heartbeat` request in `controller.capnp`. The heartbeat listener (§9.0)
|
||||
populates this field from `c->ha.my_priority`. `ha_check_heartbeat_nocluster()`
|
||||
(ha.c) compares `sender_priority` against `ha->my_priority` when `term == 0`:
|
||||
if the sender has higher priority (lower index in `--peers`), the local node
|
||||
steps down to standby. The data source is the heartbeat message, not `ha_state_t`,
|
||||
so both the schema change and the handler change are part of this design.
|
||||
|
||||
The effective split-brain window is therefore the time for the
|
||||
secondary-primary to complete Phase 2 and start processing heartbeats: ~4–13s
|
||||
per §8.5. During this window both self-declared primaries have `iomgr_init` and
|
||||
`endpoint_init` complete (per the Phase 2 ordering in §7), so both are accepting
|
||||
client connections before `ha_promote` resolves the split-brain. Conflicting
|
||||
client operations (job submissions, queue modifications) submitted to both
|
||||
primaries during this window may diverge. This falls within the parent design's
|
||||
acknowledged `split_brain_risk_accepted` model, with the caveat that the
|
||||
window is longer than a single heartbeat interval during cold start.
|
||||
|
||||
**Data divergence during split-brain**: The losing primary's WAL is discarded when it
|
||||
steps down to standby — any client operations it accepted during the split-brain
|
||||
window are lost. This design does not add a reconciliation protocol; the scope is
|
||||
bounded to the existing HA model. Operators who cannot tolerate any data loss during
|
||||
cold start should not submit client operations until the primary's `INFO("Promoted
|
||||
to primary")` log message confirms Phase 2 is complete. The typical cold-start
|
||||
window occurs only during initial cluster boot or full-cluster restart — not during
|
||||
normal single-node failover where an existing primary with a term > 0 prevents the
|
||||
split-brain scenario from occurring.
|
||||
|
||||
**PRNG**: The jitter calculation uses an xorshift64 PRNG seeded from
|
||||
`/dev/urandom`. The PRNG state is stored in `ha_state_t.jitter_seed` (one per
|
||||
node, initialised during Phase 1). The xorshift64 implementation should be a
|
||||
shared inline function in `ha.h` (or a small `prng.h` utility header) so both
|
||||
the election probe (§9.1) and the heartbeat listener (parent HA design) use the
|
||||
same generator without code duplication. The seed is initialised from
|
||||
`/dev/urandom` (read 8 bytes) during Phase 1.
|
||||
If `/dev/urandom` is unavailable, fall back to
|
||||
`clock_gettime(CLOCK_MONOTONIC, &ts)` nanoseconds XOR'd with
|
||||
`((uint64_t)priority << 56) ^ ((uint64_t)getpid() << 16)` with a `WARN` log.
|
||||
The shift mixing avoids the correlation problem where two systemd-started
|
||||
processes in the same nanosecond produce jitter values differing only by
|
||||
their priority index (a small integer). The `time(NULL) ^ getpid()` fallback
|
||||
is rejected because it produces nearly identical values for processes started
|
||||
simultaneously by systemd (same second, sequential PIDs) — exactly the scenario
|
||||
where symmetry breaking matters most.
|
||||
|
||||
### 9.1 Algorithm (design doc §5.4)
|
||||
|
||||
```c
|
||||
int am_i_primary(cluster_t *c)
|
||||
{
|
||||
/* Priority 0: unconditionally primary — highest-priority node does not probe.
|
||||
* Safe because all controllers receive the same --peers list, so ordering is
|
||||
* deterministic and priority-0 is unique across the cluster. */
|
||||
if (c->ha.my_priority == 0) return 1;
|
||||
|
||||
/* Launch one pthread per candidate with priority < my_priority.
|
||||
* Each thread makes a non-blocking connect + poll loop to the candidate,
|
||||
* sends OP_heartbeat. If ANY thread gets a live ACK before the shared
|
||||
* election_timeout_ms deadline → return 0 (standby).
|
||||
* If the deadline expires with no ACK → return 1 (primary). */
|
||||
|
||||
atomic_int election_done = 0;
|
||||
probe_ctx_t probes[MAX_HA_CANDIDATES];
|
||||
int n_probes = 0;
|
||||
|
||||
for (int i = 0; i < c->ha.my_priority; i++) {
|
||||
probes[n_probes++] = (probe_ctx_t){
|
||||
.candidate = &c->myconf->peers_cfg.candidates[i],
|
||||
.ssl_ctx = c->ha.ssl_ctx, /* stashed in Phase 1 */
|
||||
.timeout_ms = c->ha.config.probe_connect_timeout_ms,
|
||||
.election_done = &election_done,
|
||||
.got_ack = 0, /* set to 1 by probe_thread on success */
|
||||
};
|
||||
pthread_create(&probes[n_probes-1].thread, NULL,
|
||||
probe_thread, &probes[n_probes-1]);
|
||||
}
|
||||
|
||||
/* Wait for first ACK or deadline (with jitter for simultaneous-boot, see §9.0) */
|
||||
long to_ms = c->ha.config.election_timeout_ms;
|
||||
to_ms += (long)c->ha.my_priority * c->ha.config.election_priority_step_ms;
|
||||
to_ms += (long)(xorshift64_next(&c->ha.jitter_seed) % c->ha.config.election_jitter_max_ms);
|
||||
struct timespec deadline;
|
||||
clock_gettime(CLOCK_MONOTONIC, &deadline);
|
||||
deadline.tv_sec += to_ms / 1000;
|
||||
deadline.tv_nsec += (to_ms % 1000) * 1000000L;
|
||||
if (deadline.tv_nsec >= 1000000000L) {
|
||||
deadline.tv_sec++;
|
||||
deadline.tv_nsec -= 1000000000L;
|
||||
}
|
||||
int got_ack = wait_any_probe(probes, n_probes, &deadline);
|
||||
|
||||
/* Signal all threads to stop (no pthread_cancel — unsafe through OpenSSL) */
|
||||
atomic_store(&election_done, 1);
|
||||
for (int i = 0; i < n_probes; i++)
|
||||
pthread_join(probes[i].thread, NULL);
|
||||
|
||||
return got_ack ? 0 : 1;
|
||||
}
|
||||
```
|
||||
|
||||
### 9.2 Thread coordination
|
||||
|
||||
`pthread_cancel()` is **NOT** used (cancellation through OpenSSL shutdown handlers
|
||||
can deadlock). Each probe thread uses **non-blocking `connect()` + `poll()`**:
|
||||
|
||||
1. `socket()` + `fcntl(O_NONBLOCK)` + `connect()` → expect `EINPROGRESS`
|
||||
2. `poll()` with timeout `probe_connect_timeout_ms` (default 500ms)
|
||||
3. Between poll attempts, check `atomic_load(&election_done)` — if set, exit cleanly
|
||||
4. If `poll()` returns -1 and `errno != EINTR`, treat as probe failure — log at TRACE
|
||||
level and exit thread with `*got_ack = 0`
|
||||
5. After connect, `SSL_connect()` with `SSL_ERROR_WANT_READ`/`WANT_WRITE` polling;
|
||||
on hard SSL failures (certificate expired, protocol mismatch, peer disconnect),
|
||||
treat as probe failure — log at TRACE level and exit thread
|
||||
6. If `send()` or `recv()` fails on the SSL socket, treat as probe failure
|
||||
7. Send `OP_heartbeat`, receive ACK, set `*got_ack` via shared pointer
|
||||
|
||||
The per-attempt poll timeout bounds the worst-case exit delay to one poll interval
|
||||
per thread (~500ms). TCP connect timeout (kernel SYN retry) is not reached because
|
||||
the thread checks `election_done` between each poll iteration and exits early.
|
||||
Probe failures are expected when peers are down — only TRACE-level logging is
|
||||
appropriate; ERROR logs would generate noise during normal operation.
|
||||
|
||||
### 9.3 Election outcome logging
|
||||
|
||||
```
|
||||
/* Higher-priority candidate alive: */
|
||||
INFO("HA election: candidate %s:%u responded — starting as standby (my_priority=%d)",
|
||||
winner_host, winner_port, my_priority)
|
||||
|
||||
/* No higher-priority candidate alive: */
|
||||
INFO("HA election: no higher-priority candidate responded within %ums — "
|
||||
"starting as primary (my_priority=%d)",
|
||||
election_timeout_ms, my_priority)
|
||||
```
|
||||
> **SUPERSEDED.** This section (originally: the pthread-based priority
|
||||
> election — `am_i_primary()`, `probe_thread()`, `wait_any_probe()` — plus the
|
||||
> pre-iomgr `start_heartbeat_listener()`/`stop_heartbeat_listener()` heartbeat
|
||||
> listener and the `ha_check_heartbeat_nocluster()` priority-based step-down
|
||||
> used together for simultaneous-boot split-brain avoidance) described a
|
||||
> design that was later replaced. Every function named above has been deleted
|
||||
> from the codebase — see the deletion comments in `src/controller/ha.c`
|
||||
> (`am_i_primary() deleted — replaced by ha_start_election_async
|
||||
> (election-iomgr-refactor)`; `start_heartbeat_listener() / stop_heartbeat_listener()
|
||||
> deleted — never called`; `ha_check_heartbeat_nocluster() / ha_watchdog_thread() /
|
||||
> ha_start_watchdog() / ha_stop_watchdog() deleted — watchdog pthread replaced
|
||||
> by ha_arm_heartbeat_timeout_vtask`) and `src/controller/ha.h`. Election is
|
||||
> now a Raft-style quorum protocol (`ha_start_election_async()` →
|
||||
> `raft_start_election()`) driving a 4-state FSM (`CTRL_INIT` /
|
||||
> `CTRL_ELECTING` / `CTRL_PRIMARY` / `CTRL_STANDBY`), with `EPOLLRDHUP`-based
|
||||
> failover detection replacing the heartbeat-timeout watchdog pthread. See
|
||||
> "src/docs/config-replication-design.md" findings and the closed specs in
|
||||
> `.kiro/specs/` — `ha-config-bootstrap`, `election-iomgr-refactor`,
|
||||
> `raft-library`, `raft-ha-core-safety`, `ha-prevote` — for what shipped
|
||||
> instead.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -2098,16 +1873,16 @@ See `FUTURE_WORK.md`.
|
|||
|
||||
| File | Change | Complexity |
|
||||
|---|---|---|
|
||||
| `src/schema/controller.capnp` | Add `replConfig @29`, `replConfigPush @30`, `upgradeConfig @31` to `OPCode`; add `ReplConfigRequest`, `ReplConfigReply`, `ReplConfigPushRequest`, `ReplConfigPushReply`, `UpgradeConfigRequest`, `UpgradeConfigReply` structs; add `configGeneration @20 :UInt64` to `ClusterConfig` (§4.4); add `sender_priority @0 :UInt32` to `Heartbeat` struct (populated by heartbeat listener, compared by `ha_check_heartbeat_nocluster()` — hard prerequisite for simultaneous-boot symmetry, §9.0); add union arms to `Request` and `RequestReply` | Low |
|
||||
| `src/schema/controller.capnp` | Add `replConfig @29`, `replConfigPush @30`, `upgradeConfig @31` to `OPCode`; add `ReplConfigRequest`, `ReplConfigReply`, `ReplConfigPushRequest`, `ReplConfigPushReply`, `UpgradeConfigRequest`, `UpgradeConfigReply` structs; add `configGeneration @12 :UInt64` to `ClusterConfig` (§4.4; implemented at field number @12, not the @20 originally planned at design time); add `sender_priority @0 :UInt32` to `Heartbeat` struct (populated by heartbeat listener, compared by `ha_check_heartbeat_nocluster()` — hard prerequisite for simultaneous-boot symmetry, §9.0); add union arms to `Request` and `RequestReply` | Low |
|
||||
| `src/controller/main.c` | Update `conf_file` resolution: read `conf/current` pointer file, then open named versioned file (replaces flat `conf/current_config` path); remove stack-allocated `char conf_file[]` and the `myconf.conffile = conf_file` assignment — `conffile` is now heap-allocated via `asprintf()` in `load_primary_config()` or `fetch_config_from_primary()` and freed in `myconf_fini()`; add `--conf-stale-timeout` and `--conf-keep-n` CLI flag parsing; add `--peers` validation (WARN on empty host or port 0) | Trivial |
|
||||
| `src/include/myconf.h` | Add `config_version_tag` (uint64_t), `config_sha256[32]`, `conf_keep_n`, `config_generation` (uint64_t, §4.4), `conf_stale_timeout` (time_t, default 86400), and `config_stale` (bool) fields to `myconf_t` | Trivial |
|
||||
| `src/include/xce/cluster_def.h` | Add `restart_requested` (atomic_int, for config-change restart), `heartbeat_listener_tid`, `heartbeat_listener_running`, and `lost_and_found_job_count` (atomic_int, incremented during `persist_recover()` when jobs are routed to the synthetic queue, §6.5) fields to `cluster_t` | Trivial |
|
||||
| `src/controller/myconf.c` | Replace `get_default_conf()` with `load_primary_config()` (fail-fast + SHA-256 compute on primary path); config save/trim/validation functions moved to `config_repl.c`; add `myconf_fini()` to free `conffile` (now heap-allocated via `asprintf()` per §3.2), `cluster_config`, `config_version_tag`, and `config_sha256` | Low |
|
||||
| `src/controller/cluster.c` | Split `cluster_init()` into phase 1 and phase 2; gate config load on role; move `iomgr_init` and `endpoint_init` to phase 2; add `create_lost_and_found()` call before `persist_recover`; add `OP_replConfig`, `OP_replConfigPush`, and `OP_upgradeConfig` cases to `myreq_handler()` opcode dispatch; `cluster_run()` event loop checks `c->restart_requested` and exits cleanly when set; `cluster_fini()` joins heartbeat listener thread on any exit path and frees `standby_endpoints` | Medium |
|
||||
| `src/controller/myapp.c` | Restructure `myapp_run()`: call phase 1, then `am_i_primary()`, then config load/fetch, then phase 2. When `cluster_run()` returns cleanly (restart_requested=1), `myapp_run()` returns 0 to main() which exits; the systemd service manager restarts the process. Call `myconf_fini()` in the cleanup path before `cluster_fini()` | Medium |
|
||||
| `src/controller/config_repl.c` | **Phase 1**: `conf_read_current_filename()` (read HEAD-style pointer), `conf_read_raw()` (read + size-check, enforcing `CONF_MAX_SIZE`), `conf_save_config()` (write + fsync + thread-unique pointer-file swap + trim), `conf_trim_old_versions()`, `compute_config_sha256()`, `check_conf_stale()` (mtime-based staleness check, §10.3), `sanitise_cn()` (cert CN log sanitisation, §6.4), `config_contains_reserved_name()` (lost_and_found name collision check, §7.1). **Phase 2**: `handle_upgrade_config()` (SHA-256 verify, Cap'n Proto decode, generation increment §4.4, re-encode with gen+1, save, push), `push_config_to_standbys()`, `create_lost_and_found()` (synthetic queue + host, §6.5), `resolve_queue_or_lost_and_found()`, `resolve_host_or_lost_and_found()`, `clear_resource_and_pend()`. Compile-time gate: `CONF_ENABLE_UPGRADE_CONFIG` (default 0, §5.3) | Medium |
|
||||
| `src/controller/config_repl.c` | **Phase 1**: `conf_read_current_filename()` (read HEAD-style pointer), `conf_read_raw()` (read + size-check, enforcing `CONF_MAX_SIZE`), `conf_save_config()` (write + fsync + thread-unique pointer-file swap + trim), `conf_trim_old_versions()`, `compute_config_sha256()`, `check_conf_stale()` (mtime-based staleness check, §10.3), `sanitise_cn()` (cert CN log sanitisation, §6.4), `config_contains_reserved_name()` (lost_and_found name collision check, §7.1). **Phase 2**: `handle_upgrade_config()` (SHA-256 verify, Cap'n Proto decode, generation increment §4.4, re-encode with gen+1, save, push), `push_config_to_standbys()`, `create_lost_and_found()` (synthetic queue + host, §6.5), `resolve_queue_or_lost_and_found()`, `resolve_host_or_lost_and_found()`, `clear_resource_and_pend()`. Compile-time gate: `CONF_ENABLE_UPGRADE_CONFIG` (default 0 as designed; ships hardcoded to 1, §5.3 drift note) | Medium |
|
||||
| `src/include/byteorder.h` | **New file**: `be64toh()` fallback macro for non-glibc or pre-2.25 targets (§6.4 portability note). The Meson build checks for `be64toh()` at configure time; if unavailable, this header provides a portable implementation via `__builtin_bswap64()` or `le64toh()`/`ntohll()` | Trivial |
|
||||
| `src/controller/meson.build` | Add `-DCONF_PUSH_TIMEOUT_MS=500` and `-DCONF_ENABLE_UPGRADE_CONFIG=0` compile definitions (defaults, overridable via Meson options); add `config_repl.c`, `config_repl.h`, `byteorder.h` to sources | Trivial |
|
||||
| `src/controller/meson.build` | Add `-DCONF_PUSH_TIMEOUT_MS=500` and `-DCONF_ENABLE_UPGRADE_CONFIG=0` compile definitions (defaults, overridable via Meson options); add `config_repl.c`, `config_repl.h`, `byteorder.h` to sources. **Not implemented as designed**: both macros are defined via `#ifndef`/`#define` fallback directly in `config_repl.h` instead (with the enable flag hardcoded to `1`) — `meson.build` has no `-Denable-upgrade-config` option and no override of either macro. | Trivial |
|
||||
| `src/controller/ha.c` | Replace `am_i_primary()` stub with real election probing; add `probe_thread()`, `wait_any_probe()`; add `start_heartbeat_listener()` / `stop_heartbeat_listener()`; add `xorshift64_next()` (or use shared `ha.h` inline); enhance `ha_check_heartbeat_nocluster()` with priority comparison for simultaneous-boot symmetry breaking (§9.0) | High |
|
||||
| `src/controller/ha.h` | Add `ssl_ctx` field to `ha_state_t` (use `void*` to avoid pulling in `<openssl/ssl.h>` — cast to `SSL_CTX*` at use sites in election probe and heartbeat listener); add `jitter_seed` (uint64_t, for xorshift64 PRNG) and `xorshift64_next()` inline function to `ha_state_t`; add `zarray_t *standby_endpoints` to `ha_state_t` for tracking connected standby controllers; add `probe_ctx_t` struct; add `election_jitter_max_ms` and `election_priority_step_ms` to `ha_config_t` with defaults `HA_DEFAULT_ELECTION_JITTER_MAX_MS 200` and `HA_DEFAULT_ELECTION_PRIORITY_STEP_MS 500` (§9.0 priority-based base delay); update "no new TLS" comment to note SSL transport is sufficient for trusted-LAN model; update `am_i_primary()` declaration; declare `ha_is_primary()` for runtime role check (§10.12) | Low |
|
||||
| `src/controller/join.c` | Update `handle_controller_join()`: remove `-ENOSYS` (controller-controller sessions are valid — config exchange is dispatched from `myreq_handler()`); register connected standby endpoints in `c->ha.standby_endpoints` after successful join handshake; handle endpoint removal on disconnect | Low |
|
||||
|
|
@ -2123,7 +1898,8 @@ Note: the existing `src/controller/repl.c` (WAL replication — standby slot man
|
|||
ACK tracking, catch-up) is a separate module with a similar name but different domain.
|
||||
`repl.c` uses `replicator_t.standbys[]` for per-connection WAL ACK tracking;
|
||||
`config_repl.c` uses `ha_state_t.standby_endpoints` for config push fan-out.
|
||||
See `src/controller/repl.c` (340 lines) for the existing WAL replication module.
|
||||
See `src/controller/repl.c` (881 lines as of this audit, up from 340 at design time)
|
||||
for the existing WAL replication module.
|
||||
|
||||
**Total estimated LOC**: ~510 net new lines across all files (updated from ~480 to
|
||||
account for compile-time gate, `sanitise_cn()`, `config_contains_reserved_name()`,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue