docs: fix 9 P0 and 6 P1 issues from second adversarial review
P0 fixes (9): - R-22: COW vstore find-slot-and-write not atomic - R-23: vput grow path without mutex during COW - R-24: Post-growth slot double-claimed by concurrent threads - R-25: Missing key_snapshot field for checkpoint serialization - R-26: malloc(0) for WAL_CHECKPOINT entries - R-27: Fallback to older snapshot with trimmed WAL - R-28: hdr.term > ha->term + 1 too restrictive - R-29: _Atomic on enum type (implementation-defined) - R-30: handle_jobsubmit() ignores hook return value - R-31: malloc NULL check missing in standby receiver P1 fixes (5): - R-32: No heartbeat during persist_recover() on new primary - R-33: Catch-up from dead primary removed from promotion - R-34: Added total_wal_bytes field to persist_t Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
parent
54bbfa5865
commit
f1641b1fbf
1 changed files with 111 additions and 44 deletions
|
|
@ -226,7 +226,7 @@ On startup, `cluster_init()` calls `persist_recover()`:
|
|||
|
||||
1. **Find latest snapshot**: Scan `snap_path/` for the snapshot with the highest LSN
|
||||
2. **Load snapshot**: Read the snapshot file into memory
|
||||
2a. **Verify snapshot checksums**: (i) Recompute CRC32 over the header fields (lsn, timestamp, term, version, nJobs, nQueues, nWorkers) and compare against `SnapshotHeader.headerChecksum`. If mismatch, the header has been tampered or corrupted — log an error and try the next-oldest snapshot. (ii) Recompute CRC32 over the Cap'n Proto payload and compare against `SnapshotHeader.checksum`. If mismatch, log an error and try the next-oldest snapshot. If **all** snapshots fail verification and the WAL contains entries that predate any valid snapshot (i.e., WAL segments have already been trimmed), the controller **must abort with a fatal error** — silently booting with empty state would silently discard live job/worker state and is data loss. The fatal error message must include: the number of corrupt snapshots found, the LSN range of surviving WAL segments, and the instruction `"Contact support or restore from backup before restarting"`. Do not fall back to empty state in this scenario. The empty-state fallback is only valid on first boot (see below). **Snapshot version check**: After verifying both checksums, check `SnapshotHeader.version`. If the version is higher than the current controller's `WAL_FORMAT_VERSION` (defined in `persist.h` as `1`), log a fatal error and abort — a snapshot written by a newer controller cannot be safely loaded by an older one. This catches version mismatches earlier than encountering unknown WAL entry types during replay.
|
||||
2a. **Verify snapshot checksums**: (i) Recompute CRC32 over the header fields (lsn, timestamp, term, version, nJobs, nQueues, nWorkers) and compare against `SnapshotHeader.headerChecksum`. If mismatch, the header has been tampered or corrupted — log an error and try the next-oldest snapshot. (ii) Recompute CRC32 over the Cap'n Proto payload and compare against `SnapshotHeader.checksum`. If mismatch, log an error and try the next-oldest snapshot. If **all** snapshots fail verification and the WAL contains entries that predate any valid snapshot (i.e., WAL segments have already been trimmed), the controller **must abort with a fatal error** — silently booting with empty state would silently discard live job/worker state and is data loss. The fatal error message must include: the number of corrupt snapshots found, the LSN range of surviving WAL segments, and the instruction `"Contact support or restore from backup before restarting"`. Do not fall back to empty state in this scenario. The empty-state fallback is only valid on first boot (see below). **Snapshot version check**: After verifying both checksums, check `SnapshotHeader.version`. If the version is higher than the current controller's `WAL_FORMAT_VERSION` (defined in `persist.h` as `1`), log a fatal error and abort — a snapshot written by a newer controller cannot be safely loaded by an older one. This catches version mismatches earlier than encountering unknown WAL entry types during replay. **Fallback WAL verification**: When falling back from the latest snapshot to an older snapshot (due to checksum failure), the recovery code must first verify that WAL segments exist from `older_snapshot.lsn + 1` forward. If WAL segments have already been trimmed past the older snapshot's LSN (i.e., only segments covering `[latest_snapshot.lsn, ...]` exist), the fallback cannot replay the gap and the controller must abort with a fatal error. Do NOT silently accept an older snapshot with no WAL to replay — this would start the controller with stale state, silently discarding all changes between the older snapshot and the trimmed WAL.
|
||||
2b. **Deserialize**: Deserialize Cap'n Proto data, rebuild kvmgr instances. Set `capnp::ReaderOptions.traversalLimitInWords` to a **fixed upper bound** (`MAX_SNAPSHOT_BYTES / 8` words) to prevent O(n²) traversal amplification from a crafted snapshot — this is the same hardening practice used for WAL payload deserialization (Part 6.4). Do NOT set it proportional to the actual file size, as a small file with deeply nested structure can still trigger excessive traversal. The limit should be `64 * 1024 * 1024 / 8 = 8M words` (the 64 MB is the expected max snapshot size). If the snapshot file is larger than `MAX_SNAPSHOT_BYTES` (default: 256 MB — 4× the expected 64 MB for 10K jobs × 1 KB + overhead), log an `ERROR` and reject the snapshot.
|
||||
- Jobs → `makeJob(c->jobmgr, ...)` (reuse existing `create_job()`)
|
||||
- Workers → `restore_worker(c, host)` (already exists in `worker.c:367`)
|
||||
|
|
@ -467,7 +467,7 @@ Since indices are stable (hash table maps key→same index always), a `vput(42,
|
|||
|
||||
This is required on ALL architectures, not just ARM — compiler reordering is sufficient to cause the race even on x86-64. The kvmgr READ lock in `vget()` callers provides happens-before between `vput()` and `vget()` on the **same vcpu thread** (serialized by lock/unlock), but does not prevent the compiler from reordering stores within a single `vput()` call. The atomic store on `.inuse` is the only reliable barrier against both compiler and CPU reordering.
|
||||
|
||||
**Concurrent `vstore` race under READ lock (CRITICAL)**: During COW mode, multiple vcpu threads may hold the kvmgr READ lock simultaneously and call `vstore()`. The kvmgr READ lock does NOT serialize these calls — it allows concurrent readers. Two concurrent `vstore()` calls hitting the "no free slot in `[shadow_caps, a->caps)`" path will both call `xzarray_ensure_items(a, a->caps + a->delta)`, and both will write to `a->data[old_caps]` — the second overwrites the first, silently losing data. **Fix**: During COW mode, `vstore()` must acquire the inner xzarray mutex (`mylock_lock(&priv->lock)`) around the growth-and-assign section. This is the ONLY place in the COW protocol where the inner mutex is required — normal-mode vput and vstore rely on the kvmgr WRITE lock for serialization. The growth section in `xzarray_store()` must be:
|
||||
**Concurrent `vstore` race under READ lock (CRITICAL)**: During COW mode, multiple vcpu threads may hold the kvmgr READ lock simultaneously and call `vstore()`. The kvmgr READ lock does NOT serialize these calls — it allows concurrent readers. Two concurrent `vstore()` calls can both see `inuse==0` at the same `idx` during the scan phase, and the second silently overwrites the first's `.data` — silent data loss. **Fix**: During COW mode, the entire `xzarray_store()` find-slot-and-write sequence is serialized via the inner xzarray mutex (`mylock_lock(&priv->lock)`). The mutex is acquired at the top of the COW-mode path and released on all return paths. This is the ONLY place in the COW protocol where the inner mutex is required — normal-mode vput and vstore rely on the kvmgr WRITE lock for serialization (unchanged).
|
||||
```c
|
||||
/* COW mode growth: serialize around the grow-and-assign */
|
||||
mylock_lock(&priv->lock);
|
||||
|
|
@ -509,7 +509,16 @@ static int xzarray_store(meta_obj_t* obj, void* data) {
|
|||
* inuse must be read with atomic_load_explicit(acquire) to pair
|
||||
* with vput's atomic_store_explicit(release) — both vstore and
|
||||
* vput hold the kvmgr READ lock concurrently, so the atomic
|
||||
* ordering (not the lock) provides inter-reader visibility. */
|
||||
* ordering (not the lock) provides inter-reader visibility.
|
||||
*
|
||||
* CRITICAL: The entire find-slot-and-write sequence is serialized
|
||||
* via the inner xzarray mutex (priv->lock). Without this, two
|
||||
* concurrent vstore callers can both see inuse==0 at the same idx
|
||||
* during the scan, and the second silently overwrites the first's
|
||||
* .data — a silent data loss bug. The mutex is only held during
|
||||
* COW mode; normal-mode vput/vstore rely on the kvmgr WRITE lock
|
||||
* for serialization (unchanged). */
|
||||
mylock_lock(&priv->lock);
|
||||
for (idx = a->pos; idx < (int)a->caps; idx++) {
|
||||
if (atomic_load_explicit((_Atomic uint32_t*)&a->data[idx].inuse,
|
||||
memory_order_acquire) == 0) {
|
||||
|
|
@ -518,35 +527,24 @@ static int xzarray_store(meta_obj_t* obj, void* data) {
|
|||
1, memory_order_release);
|
||||
a->count++;
|
||||
a->pos = idx + 1;
|
||||
mylock_unlock(&priv->lock);
|
||||
return idx;
|
||||
}
|
||||
}
|
||||
/* No free slot in [shadow_caps, a->caps) — grow array.
|
||||
* Must serialize around growth: concurrent vstore calls under READ lock
|
||||
* would both hit this path. See the concurrent vstore race fix above. */
|
||||
* Already holding the mutex, so no need to re-acquire. */
|
||||
int old_caps = a->caps;
|
||||
mylock_lock(&priv->lock);
|
||||
/* Re-check: another thread may have grown while we waited for the lock.
|
||||
* Check a->caps > old_caps (not a->data[old_caps].inuse) — the latter is
|
||||
* out-of-bounds when the array hasn't grown yet (index old_caps is past
|
||||
* the end), and reads garbage. The correct test is whether another thread
|
||||
* already updated a->caps past our saved value. */
|
||||
if (a->caps <= old_caps) {
|
||||
if (a->caps > SIZE_MAX - a->delta) { mylock_unlock(&priv->lock); return -ENOMEM; }
|
||||
xzarray_ensure_items(a, a->caps + a->delta);
|
||||
}
|
||||
/* After growth (by us or another thread), use the first free slot at a->caps-1
|
||||
* (the slot just before the new capacity after xzarray_ensure_items zeroed it).
|
||||
* Re-read old_caps since another thread may have grown — the new slot is always
|
||||
* at the position just before a->caps after growth. */
|
||||
int free_idx = (int)a->caps - 1;
|
||||
a->data[free_idx].data = data;
|
||||
atomic_store_explicit((_Atomic uint32_t*)&a->data[free_idx].inuse,
|
||||
if (a->caps > SIZE_MAX - a->delta) { mylock_unlock(&priv->lock); return -ENOMEM; }
|
||||
xzarray_ensure_items(a, a->caps + a->delta);
|
||||
/* Use the first new slot after growth. Since we hold the mutex,
|
||||
* no other thread can have claimed this slot. */
|
||||
a->data[old_caps].data = data;
|
||||
atomic_store_explicit((_Atomic uint32_t*)&a->data[old_caps].inuse,
|
||||
1, memory_order_release);
|
||||
a->count++;
|
||||
a->pos = free_idx + 1;
|
||||
a->pos = old_caps + 1;
|
||||
mylock_unlock(&priv->lock);
|
||||
return free_idx;
|
||||
return old_caps;
|
||||
}
|
||||
|
||||
/* Normal path: unchanged circular scan */
|
||||
|
|
@ -1353,9 +1351,10 @@ When a standby promotes to primary:
|
|||
|
||||
1. **Stop WAL replay** from old primary
|
||||
2. **Increment term** (`ha.term++`) — fencing token for stale standbys
|
||||
3. **Flush the local WAL** (`persist_flush()` — fsync the last WAL segment to ensure all replicated entries are on disk) and **verify WAL consistency** (validate CRC chain from the standby's own last snapshot LSN forward — standbys run `persist_recover()` on startup and maintain their own local WAL and snapshots, so the validation starting point is the standby's `persist_t.checkpoint_lsn`). Do **NOT** call `persist_recover()` again — the standby's kvmgrs are already populated from replication (via `apply_wal_entry()` during normal `OP_replEntry` processing) AND from the startup recovery pass. Calling `persist_recover()` a second time would double-apply entries. Only a WAL flush + CRC chain validation is needed. If the standby has a WAL gap (entries the old primary committed but had not yet replicated before crashing), the promotion path requests missing entries from the old primary's WAL on disk via catch-up (see Part 7 catch-up protocol). **WAL consistency guarantee**: the standby's local WAL is a strict prefix of the old primary's WAL — standbys only write entries received via `OP_replEntry` from the primary. No divergence is possible if the standby was continuously connected; the standby's WAL cannot contain entries the old primary did not originate. The only split-brain divergence scenario (standby accepted writes from clients during a partition) cannot occur because standbys do not accept writes — they only redirect to the primary.
|
||||
3. **Flush the local WAL** (`persist_flush()` — fsync the last WAL segment to ensure all replicated entries are on disk) and **verify WAL consistency** (validate CRC chain from the standby's own last snapshot LSN forward — standbys run `persist_recover()` on startup and maintain their own local WAL and snapshots, so the validation starting point is the standby's `persist_t.checkpoint_lsn`). Do **NOT** call `persist_recover()` again — the standby's kvmgrs are already populated from replication (via `apply_wal_entry()` during normal `OP_replEntry` processing) AND from the startup recovery pass. Calling `persist_recover()` a second time would double-apply entries. Only a WAL flush + CRC chain validation is needed. **WAL gap handling**: Async-replicated entries that the old primary committed but had not yet replicated before crashing are **lost** — the dead primary cannot serve catch-up. This is a known limitation of the async replication model: non-sync WAL entries (job starts, finishes, rusage updates) may be lost on failover. The promoted primary's view is consistent but potentially stale; async state gaps self-heal through worker reconnection (see step 9). **WAL consistency guarantee**: the standby's local WAL is a strict prefix of the old primary's WAL — standbys only write entries received via `OP_replEntry` from the primary. No divergence is possible if the standby was continuously connected; the standby's WAL cannot contain entries the old primary did not originate. The only split-brain divergence scenario (standby accepted writes from clients during a partition) cannot occur because standbys do not accept writes — they only redirect to the primary.
|
||||
4. **Switch role** to `CTRL_PRIMARY` — only after recovery succeeds. **If recovery fails, abort this promotion attempt and set `ha.promotion_failed = 1`**. The standby remains in `CTRL_STANDBY` role, but since the old primary is dead (that triggered this promotion), the next heartbeat watchdog check (`ha_check_heartbeat()`) will re-fire `ha_promote_to_primary()`. The `promotion_failed` flag gates retry: the standby retries promotion on each watchdog tick unless: (a) a new heartbeat is received from a higher-priority candidate (another controller won the race), or (b) a configurable `max_promotion_retries` (default: 5) is exceeded, at which point the standby logs `FATAL "cannot promote after %d attempts — manual intervention required"` and stops retrying. This prevents a permanent no-primary state while bounding the retry window.
|
||||
5. **Register persistence hooks** — standby did not register persistence hooks during startup (hooks are registered only after `persist_recover()` completes, and only on the primary path; standbys register them here on promotion). **Idempotent registration guard**: `ha_promote_to_primary()` must check whether hooks are already registered before adding them (e.g., via a `persist->hooks_registered` flag). Without this guard, a promote→demote→re-promote cycle would accumulate duplicate hooks that fire N times per event, causing double WAL writes and replication sends.
|
||||
5.5 **Start heartbeat immediately** — begin sending heartbeat messages to other standbys BEFORE running `persist_recover()`. The promoted standby must establish its authority as the new primary by sending heartbeats with its incremented term. This prevents other standbys from timing out and promoting themselves during the potentially lengthy `persist_recover()` call (10–30 seconds for large state). The heartbeat thread is launched with the new term and role set to PRIMARY, but `cluster_start()` (step 8) is deferred until recovery completes. **CRITICAL**: The heartbeat messages sent during this pre-recovery phase contain only term and role — no LSN or state information. Other standbys receiving these heartbeats update their `ha.term` and `primary_host` to the new primary, but do not expect any replication data until step 8 is reached.
|
||||
6. **Initialize replicator** — establish connections to remaining standbys
|
||||
7. **Gate the scheduler**: pause or flush the scheduler's run queue before accepting writes. The scheduler maintains allocation state derived from WAL-replayed job state; async-replicated events (job starts/finishes) may be stale immediately after promotion. The scheduler must re-query job state from the promoted primary before making scheduling decisions (see step 8).
|
||||
8. **Start normal primary operation** — `cluster_start()`, accept connections
|
||||
|
|
@ -1542,24 +1541,44 @@ static void* standby_recv_thread(void* arg)
|
|||
ERROR("replication: oversized payload %u > MAX_WAL_PAYLOAD_BYTES", hdr.payload_len);
|
||||
break; /* Treat as protocol error — drop connection */
|
||||
}
|
||||
payload = malloc(hdr.payload_len);
|
||||
if (read(ha->primary_ep->fd, payload, hdr.payload_len) != hdr.payload_len)
|
||||
break;
|
||||
/* Guard against malloc(0) for WAL_CHECKPOINT entries (payload_len=0).
|
||||
* For zero-payload entries, skip malloc entirely — no read or free needed.
|
||||
* For non-zero payloads, always check for OOM. */
|
||||
payload = NULL;
|
||||
if (hdr.payload_len > 0) {
|
||||
payload = malloc(hdr.payload_len);
|
||||
if (payload == NULL) {
|
||||
ERROR("replication: OOM allocating %u bytes for lsn=%lu",
|
||||
hdr.payload_len, hdr.lsn);
|
||||
break; /* Disconnect — cannot safely process entry */
|
||||
}
|
||||
if (read(ha->primary_ep->fd, payload, hdr.payload_len) != hdr.payload_len) {
|
||||
free(payload);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
/* Validate term — reject stale or implausible entries */
|
||||
/* Validate term — reject stale entries but allow multi-promotion jumps.
|
||||
* A primary term lower than standby's term means the standby has a
|
||||
* higher term (possibly from a previous self-promotion or a different
|
||||
* primary) — reject the entry, do not advance applied_lsn, and
|
||||
* continue processing. The standby should not follow a lower-term primary. */
|
||||
if (hdr.term < ha->term) {
|
||||
free(payload);
|
||||
continue;
|
||||
}
|
||||
/* Accept any term >= ha->term. During a multi-promotion partition, the
|
||||
* primary's term may have advanced by more than 1 relative to the
|
||||
* standby's ha->term (e.g., multiple promotions while the standby was
|
||||
* isolated). The old check hdr.term > ha->term + 1 would permanently
|
||||
* disconnect the standby from a legitimate primary that advanced
|
||||
* multiple terms. Instead, if the term jumped more than +1, update
|
||||
* ha->term to match the primary's term — the primary is the
|
||||
* authoritative term leader. */
|
||||
if (hdr.term > ha->term + 1) {
|
||||
/* Term is too high — a valid primary should only ever send entries
|
||||
* with term == ha->term (steady state) or term == ha->term + 1
|
||||
* (after promotion of this standby or another candidate). Higher
|
||||
* terms indicate a protocol error or tampering. */
|
||||
ERROR("replication: implausible term %lu (expected <= %lu) — "
|
||||
"protocol error or tampering", hdr.term, ha->term + 1);
|
||||
free(payload);
|
||||
break; /* Tear down connection — primary may be compromised */
|
||||
WARN("replication: term jump from %lu to %lu (multi-promotion partition heal)",
|
||||
ha->term, hdr.term);
|
||||
ha->term = hdr.term;
|
||||
}
|
||||
|
||||
/* Validate LSN sequence — entries must arrive in strictly increasing order.
|
||||
|
|
@ -2016,8 +2035,25 @@ if (jobid == 0) {
|
|||
}
|
||||
|
||||
/* Collect jobs into list... */
|
||||
xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW, c, &jlist);
|
||||
/* ↑ SYNC replication happens inside this hook call */
|
||||
int hook_rc = xce_run_hooks(AFTER_EVENT, HOOK_DETAIL_JOB_NEW, c, &jlist);
|
||||
/* ↑ SYNC replication happens inside this hook call.
|
||||
* hook_rc != 0 means the persistence hook failed.
|
||||
* -EIO: local WAL write failed (disk full, permanent — abort submit)
|
||||
* -ETIMEDOUT: wrote locally but no standby ACKed (transient — degraded durability) */
|
||||
if (hook_rc != 0) {
|
||||
if (hook_rc == -EIO) {
|
||||
/* Permanent WAL failure — do not ACK to client.
|
||||
* The job exists in-memory but is not durable. */
|
||||
reply.status = -EIO;
|
||||
reply.data_which = RequestReply_data_nodata;
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
ERROR("job submit: persistence hook failed with -EIO (local WAL write error) — rejecting submit");
|
||||
return -EIO;
|
||||
}
|
||||
/* hook_rc == -ETIMEDOUT: locally durable but no standby ACK.
|
||||
* Still reply with success but log the degraded durability. */
|
||||
WARN("job submit: persistence hook returned -ETIMEDOUT (no standby ACK within timeout) — accepting with degraded durability");
|
||||
}
|
||||
|
||||
/* Reply AFTER hooks — client ACK implies durability */
|
||||
reply.status = 0;
|
||||
|
|
@ -2795,6 +2831,19 @@ The following issues were identified during multi-perspective review and resolve
|
|||
| R-19 | Replicated entry LSN handling unspecified (P1) | Added `persist_wal_replicated_entry()` function — writes primary's LSN/CRC directly, updates `current_lsn` and `last_crc` to match |
|
||||
| R-20 | `seg_offset` in-memory and lost on crash — recovery trust boundary undefined (P1) | Clarified: recovery scans full segment file using CRC validity, not `seg_offset` — `seg_offset` is runtime-only for fsync tracking |
|
||||
| R-21 | WAL segment rotation `prev_crc == WAL_CRC_NONE` handling during recovery unspecified (P1) | Documented: `WAL_CRC_NONE` at segment-aligned position is valid chain break, not corruption |
|
||||
| R-22 | COW vstore find-slot-and-write not atomic — concurrent slot overwrite (P0) | Wrapped entire COW-mode find-slot-and-write in `mylock_lock(&priv->lock)`; simplified growth path |
|
||||
| R-23 | `vput` grow path calls `xzarray_ensure_items()` without mutex during COW (P0) | Resolved by R-22 fix — entire COW write path (including growth) is now under the inner mutex |
|
||||
| R-24 | Post-growth slot assignment `a->caps - 1` double-claimed by concurrent threads (P0) | Resolved by R-22 fix — mutex held around growth + assignment prevents double-claim |
|
||||
| R-25 | Shadow-only iteration is prose discipline, not enforced API (P0) | Added `persist_t.key_snapshot` field to capture kvtab mapping under write lock during freeze |
|
||||
| R-26 | `malloc(0)` for WAL_CHECKPOINT entries → NULL dereference (P0) | Added `payload_len > 0` guard before malloc; `apply_wal_entry()` receives NULL for zero-payload entries |
|
||||
| R-27 | Fallback to older snapshot with trimmed WAL → silent stale boot (P0) | Added WAL-existence verification before accepting older snapshot fallback |
|
||||
| R-28 | `hdr.term > ha->term + 1` permanently disconnects multi-promotion standby (P0) | Replaced with term-accept-and-update: primary's term is authoritative, standby updates `ha->term` to match |
|
||||
| R-29 | `_Atomic ctrl_role_t` on enum type — C11 semantics implementation-defined (P0) | Changed to `_Atomic int32_t` with cast to `ctrl_role_t` at access point |
|
||||
| R-30 | `handle_jobsubmit()` ignores `xce_run_hooks()` return value — false ACK on WAL failure (P0) | Added return value check: `-EIO` aborts submit, `-ETIMEDOUT` accepts with degraded durability warning |
|
||||
| R-31 | `malloc(hdr.payload_len)` NULL check missing in standby receiver (P0) | Added OOM check; break connection on failure |
|
||||
| R-32 | No heartbeat during `persist_recover()` on new primary → split-brain (P1) | Added step 5.5: start heartbeat before `persist_recover()` with new term, defer `cluster_start()` |
|
||||
| R-33 | Catch-up from dead primary specified in promotion step 3 (P1) | Removed catch-up-from-dead-primary — async gaps acknowledged as lost, self-heal via worker reconnect |
|
||||
| R-34 | No `total_wal_bytes` field for inline checkpoint trigger (P1) | Added `total_wal_bytes` counter to `persist_t` struct |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -2827,6 +2876,19 @@ typedef struct {
|
|||
MUST NOT be persist.lock (which is mylock_t) —
|
||||
pthread_cond_timedwait requires pthread_mutex_t*.
|
||||
See Part 3.8 for rationale. */
|
||||
uint64_t total_wal_bytes; /* Cumulative WAL size across all segments.
|
||||
Updated on each write, adjusted on segment
|
||||
rotation and WAL trimming. Used by the inline
|
||||
checkpoint trigger (Part 3.11) to decide when
|
||||
to checkpoint based on snap_wal_size_mb. */
|
||||
size_t* key_snapshot; /* Key→index mapping captured during freeze.
|
||||
Allocated during xzarray_freeze() under the
|
||||
write lock: one entry per live entity. Freed
|
||||
at the end of serialization (Part 3.8).
|
||||
Without this snapshot, serialization must read
|
||||
kvtab during the no-lock phase, which races with
|
||||
concurrent vput/vstore. NULL outside of
|
||||
checkpoint. See Part 3.8 for lifecycle. */
|
||||
} persist_t;
|
||||
|
||||
int persist_init(persist_t* p, persist_config_t* config, cluster_t* c);
|
||||
|
|
@ -3045,11 +3107,16 @@ int repl_health_check(replicator_t* r);
|
|||
typedef enum { CTRL_PRIMARY = 0, CTRL_STANDBY = 1 } ctrl_role_t;
|
||||
|
||||
typedef struct {
|
||||
_Atomic ctrl_role_t role; /* Atomic — ALL reads must use atomic_load_explicit(acquire).
|
||||
* vcpu threads check role without holding locks, so writes
|
||||
* (during promote/demote) use atomic_store_explicit(release).
|
||||
* If any code path reads ha.role without the acquire barrier,
|
||||
* the role-change window mitigation is incomplete. */
|
||||
_Atomic int32_t role; /* Atomic — ALL reads must use atomic_load_explicit(acquire).
|
||||
* Uses _Atomic int32_t (not _Atomic ctrl_role_t) because
|
||||
* C11 atomic semantics on enum types are implementation-defined;
|
||||
* some compilers silently ignore _Atomic on enums, falling back
|
||||
* to non-atomic loads. The role-check window mitigation (Part 5.7)
|
||||
* requires a proven lock-free atomic — _Atomic int32_t guarantees
|
||||
* this on all supported platforms. Cast to ctrl_role_t at access:
|
||||
* (ctrl_role_t)atomic_load_explicit(&c->ha.role, memory_order_acquire)
|
||||
* Writes use atomic_store_explicit with release:
|
||||
* atomic_store_explicit(&c->ha.role, (int32_t)CTRL_PRIMARY, memory_order_release) */
|
||||
int my_priority; /* My index in candidates[] */
|
||||
replicator_t repl; /* Replicate to standbys (primary) */
|
||||
uint64_t applied_lsn; /* Last applied WAL entry (standby) */
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue