forked from lei.chen/myxce
chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x), heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup. New: ha-consistency-invariants umbrella (requirements/design/research) and its four implementation sub-specs (specs generated via batch, cross-spec review applied), steering roadmap + product.md Raft+CP decision.
This commit is contained in:
parent
084244941b
commit
06135954ff
88 changed files with 25611 additions and 5285 deletions
|
|
@ -5,10 +5,9 @@ C daemon managing cluster-level job and queue operations. Acts as the central co
|
|||
|
||||
## Build
|
||||
```bash
|
||||
# From workspace root
|
||||
nix develop
|
||||
meson build && ninja -C build
|
||||
/xce-build controller # debug + ASAN + clean build + runs tests
|
||||
```
|
||||
For full build system details, see `.kiro/steering/build-deployment.md`.
|
||||
|
||||
## Architecture
|
||||
Core source files handle distinct domains:
|
||||
|
|
@ -25,7 +24,9 @@ Links against `libxce` from `src/lib/`.
|
|||
- Component-specific headers in `include/`
|
||||
- Shared types defined in `src/include/xce/` (libxce)
|
||||
|
||||
## HA Protocol Verification
|
||||
## HA Protocol
|
||||
|
||||
See `.kiro/steering/ha-async-io.md` for the full HA protocol reference: async I/O model (iomgr/vtaskmgr/delaymgr), connection ownership, Raft election, heartbeat model, two-tier replication consistency, dispatch gate, and TLA+ variable mapping. The TLA+ safety invariants and protocol verification checklist below are the authoritative controller-specific verification gates — keep them here.
|
||||
|
||||
### Raft TLA+ Formal Specification
|
||||
|
||||
|
|
@ -91,94 +92,9 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
|
|||
|
||||
## HA Design Constraints
|
||||
|
||||
### Inbound vs. Outbound Connection Ownership (CRITICAL)
|
||||
|
||||
**Strict rule: inbound connections are for voting only. All persistent control channels are outbound from the primary.**
|
||||
|
||||
| Connection type | Direction | Purpose | Who closes |
|
||||
|---|---|---|---|
|
||||
| Vote RPC | Standby → Primary (outbound from standby) | `OPCode_requestVote` + reply only | Remote closes after reply |
|
||||
| Standby control channel | Primary → Standby (outbound from primary) | ctrlhello, config push, WAL replication, heartbeat | `vote_ep_err_handler` / `standby_ep_err_handler_primary` on disconnect |
|
||||
|
||||
Rules:
|
||||
- **Inbound vote connections** (accepted on the listener) are used **exclusively** for the vote exchange. After the reply is sent the primary does NOT reuse the connection for config/replication. The inbound ep is left open until the remote closes it.
|
||||
- **Primary never uses an inbound connection to push state.** All state push (ctrlhello, config, WAL, heartbeat nop) goes over **outbound connections** stored in `ha->standby_endpoints`.
|
||||
- During election: candidate opens outgoing vote connections (one per peer). After winning, `raft_cb_become_leader` promotes surviving vote connections to `standby_endpoints`.
|
||||
- Late joiners: the primary's heartbeat task (`primary_heartbeat_fn`) scans the peers list and **opens a fresh outbound connection** to any peer absent from `standby_endpoints`. On connect, `on_standby_connected` sends ctrlhello and registers the ep.
|
||||
- `CTRL_STANDBY` must not call `ha_find_and_connect_to_primary` or schedule `fetch_config_from_primary` after an election result — the primary will connect to it.
|
||||
|
||||
### Asynchronous, Non-Blocking I/O (CRITICAL)
|
||||
All inter-controller communication uses iomgr (epoll-based) and vtaskmgr:
|
||||
- `iomgr` dispatches inbound packets to vtask handlers — never block inside a vtask.
|
||||
- `vtaskmgr` runs handlers on a thread pool; each handler must return quickly.
|
||||
- Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send buffer — non-blocking.
|
||||
- Blocking I/O (e.g., `fetch_config_from_primary`, `ha_find_and_connect_to_primary`) is only permitted from `delaymgr` worker threads, **never** from iomgr/vtask context.
|
||||
|
||||
### Heartbeat Model
|
||||
|
||||
XCE uses a bidirectional heartbeat model aligned with Raft's AppendEntries:
|
||||
- The **primary** sends periodic heartbeats (`OPCode_nop`) to all standbys via `ha_start_primary_heartbeat` (delaymgr task, interval=3s). During `CTRL_PRIMARY_COMMITTING`, this also serves as the replication driver — the primary evaluates standby state from heartbeat replies and pushes config/WAL/snapshot as needed.
|
||||
- The **standby** replies to each heartbeat with its current `(applied_lsn, config_tag)` in `RequestReply.stateReport`. This gives the primary continuous visibility into standby convergence.
|
||||
- The **standby** also arms a liveness watchdog (`ha_arm_heartbeat_timeout_vtask`) when entering `CTRL_STANDBY`. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), it triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
|
||||
- `raft.c` (the election library) is election-only. Heartbeat and replication logic belongs in the HA layer (`ha.c`, `config_repl.c`).
|
||||
|
||||
### Connection Ownership
|
||||
- Promoted vote connections (`ha->standby_endpoints`): owned by `vote_ep_err_handler`; freed when the standby disconnects.
|
||||
- Late-joiner inbound connections: owned by the cluster endpoint array (`c->endpoints`); the `ep_private_t` role identifies `HostRole_controller` for cleanup in `handle_endpoint_error`.
|
||||
- `ha->primary_ctrl_fd`: set by `handle_ctrlhello` on the standby when the first ctrlhello arrives; used by `ha_on_primary_ctrl_fd_rdhup` to trigger re-election on disconnect.
|
||||
|
||||
## HA Cluster Deployment
|
||||
|
||||
Use the `/xce-deploy-ha` skill to deploy a local 3-node HA cluster for testing:
|
||||
|
||||
```
|
||||
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
|
||||
```
|
||||
|
||||
**Prerequisites:**
|
||||
- Build binary: `/xce-build controller`
|
||||
- TLS certs and `cluster.bin` at `ha/data/` (already in repo)
|
||||
|
||||
**Quick deploy (from repo root):**
|
||||
```bash
|
||||
# Deploy + start all 3 nodes with raft trace logging
|
||||
bash src/tools/deploy-ha.sh \
|
||||
src/xce.build/controller/xce_controller \
|
||||
ha/data \
|
||||
/tmp/xce-deploy \
|
||||
"localhost:9991,localhost:9992,localhost:9993"
|
||||
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
|
||||
sleep 0.5
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &
|
||||
```
|
||||
|
||||
**Verify health (after ~15s):**
|
||||
```bash
|
||||
# Primary elected and config pushed
|
||||
grep -i "CTRL_PRIMARY\|Config committed" /tmp/xce-deploy/logs/ctrl-1.log | grep -v DEBUG3
|
||||
# Config files present on standbys
|
||||
find /tmp/xce-deploy/controller/ctrl-2/conf /tmp/xce-deploy/controller/ctrl-3/conf -type f
|
||||
```
|
||||
|
||||
**Clean restart** (keeps ctrl-1 config + all certs, wipes data/standby-conf/logs):
|
||||
```bash
|
||||
kill -9 $(pgrep xce_controller) 2>/dev/null || true
|
||||
bash /tmp/xce-deploy/scripts/cleanup.sh
|
||||
# Then restart nodes (no redeploy needed)
|
||||
VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-1 &
|
||||
sleep 0.5; VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-2 &
|
||||
sleep 0.5; VERBOSE=-vvv /tmp/xce-deploy/scripts/start.sh ctrl-3 &
|
||||
```
|
||||
|
||||
See `.claude/skills/xce-deploy-ha/SKILL.md` for full troubleshooting reference.
|
||||
Covered in `.kiro/steering/ha-async-io.md` (connection ownership, async I/O, heartbeat, two-tier replication, dispatch gate) and `.kiro/steering/build-deployment.md` (deploy commands, cluster startup/restart/verification). The TLA+ safety invariants above remain the controller-specific verification reference.
|
||||
|
||||
## Key Files
|
||||
- `src/controller/` — Controller source files
|
||||
- `include/` — Controller-specific headers
|
||||
- `src/tools/deploy-ha.sh` — HA cluster deploy script
|
||||
|
||||
## See Also
|
||||
- Shared library: `src/lib/CLAUDE.md`
|
||||
|
|
|
|||
|
|
@ -694,6 +694,17 @@ int handle_repl_config_push(cluster_t *c, endpoint_t *ep, xce_request_t *req)
|
|||
return -EINVAL;
|
||||
}
|
||||
|
||||
/* Req 1.3 (ha-heartbeat-liveness): any msg from primary resets standby
|
||||
* liveness and steps down from CTRL_ELECTING (live primary discovered). */
|
||||
{
|
||||
ctrl_role_t role = (ctrl_role_t)atomic_load_explicit(
|
||||
&c->ha.role, memory_order_acquire);
|
||||
if (role == CTRL_STANDBY)
|
||||
ha_on_heartbeat_received(&c->ha);
|
||||
else if (role == CTRL_ELECTING)
|
||||
ha_discover_primary(&c->ha, c);
|
||||
}
|
||||
|
||||
repl_config_push_req_t *rcpr = req->body.replconfigpushrequestmsg;
|
||||
|
||||
if (!rcpr->raw_bytes || rcpr->raw_bytes_len == 0 || !rcpr->raw_sha256) {
|
||||
|
|
@ -1404,6 +1415,9 @@ int (*xce_wal_stream_request_hook)(endpoint_t *ep, xce_request_t *req) = NULL;
|
|||
#endif
|
||||
|
||||
/* send_wal_batch: enqueue a single OPCode_walCatchupEntry batch message.
|
||||
* Entry payloads (raw_payload/raw_payload_len) travel as capnp Data via
|
||||
* xce_wal_batch_req_raw (task 7.3) — the generated encode path cannot
|
||||
* carry raw bytes (xce_data_t limitation, same as replConfigPush).
|
||||
* Does not free any memory — caller manages entry lifetime.
|
||||
* Returns 0 on success. */
|
||||
static int send_wal_batch(endpoint_t *ep,
|
||||
|
|
@ -1412,24 +1426,24 @@ static int send_wal_batch(endpoint_t *ep,
|
|||
if (n_entries <= 0)
|
||||
return 0;
|
||||
|
||||
/* Build the batch container entry */
|
||||
wal_catchup_entry_t batch;
|
||||
memset(&batch, 0, sizeof(batch));
|
||||
batch.n_entries = n_entries;
|
||||
batch.entries = entries;
|
||||
|
||||
xce_request_t req;
|
||||
memset(&req, 0, sizeof(req));
|
||||
req.opcode = OPCode_walCatchupEntry;
|
||||
req.body_which = Request_body_nomsg;
|
||||
req.body.walcatchupentrymsg = &batch;
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
if (xce_wal_stream_request_hook != NULL)
|
||||
if (xce_wal_stream_request_hook != NULL) {
|
||||
/* Build the batch container the hook expects */
|
||||
wal_catchup_entry_t batch;
|
||||
memset(&batch, 0, sizeof(batch));
|
||||
batch.n_entries = n_entries;
|
||||
batch.entries = entries;
|
||||
|
||||
xce_request_t req;
|
||||
memset(&req, 0, sizeof(req));
|
||||
req.opcode = OPCode_walCatchupEntry;
|
||||
req.body_which = Request_body_walcatchupentrymsg;
|
||||
req.body.walcatchupentrymsg = &batch;
|
||||
return xce_wal_stream_request_hook(ep, &req);
|
||||
}
|
||||
#endif
|
||||
if (ep)
|
||||
xce_request(ep, &req);
|
||||
xce_wal_batch_req_raw(ep, entries, n_entries);
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -1505,7 +1519,7 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
|
|||
if (batch_n > 0) {
|
||||
send_wal_batch(ep, batch_entries, batch_n);
|
||||
for (int bi = 0; bi < batch_n; bi++) {
|
||||
free(batch_entries[bi]->payload);
|
||||
free(batch_entries[bi]->raw_payload);
|
||||
free(batch_entries[bi]);
|
||||
}
|
||||
batch_n = 0;
|
||||
|
|
@ -1525,10 +1539,12 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
|
|||
if (pbuf) {
|
||||
ssize_t pr = pread(fd, pbuf, hdr.payload_len,
|
||||
(off_t)(off + sizeof(hdr)));
|
||||
if (pr == (ssize_t)hdr.payload_len)
|
||||
entry->payload = (xce_data_t *)pbuf;
|
||||
else
|
||||
if (pr == (ssize_t)hdr.payload_len) {
|
||||
entry->raw_payload = pbuf;
|
||||
entry->raw_payload_len = hdr.payload_len;
|
||||
} else {
|
||||
free(pbuf);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1541,7 +1557,7 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
|
|||
batch_bytes >= WAL_STREAM_BATCH_BYTES) {
|
||||
send_wal_batch(ep, batch_entries, batch_n);
|
||||
for (int bi = 0; bi < batch_n; bi++) {
|
||||
free(batch_entries[bi]->payload);
|
||||
free(batch_entries[bi]->raw_payload);
|
||||
free(batch_entries[bi]);
|
||||
}
|
||||
batch_n = 0;
|
||||
|
|
@ -1557,7 +1573,7 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
|
|||
if (batch_n > 0) {
|
||||
send_wal_batch(ep, batch_entries, batch_n);
|
||||
for (int bi = 0; bi < batch_n; bi++) {
|
||||
free(batch_entries[bi]->payload);
|
||||
free(batch_entries[bi]->raw_payload);
|
||||
free(batch_entries[bi]);
|
||||
}
|
||||
}
|
||||
|
|
@ -1610,17 +1626,14 @@ int ha_send_snapshot_to_standby(endpoint_t *ep, const char *snap_dir,
|
|||
uint64_t offset = 0;
|
||||
uint32_t chunk_index = 0;
|
||||
|
||||
/* Chunk the snapshot file and enqueue each chunk via xce_request().
|
||||
* Payload encoding uses NULL for now — the user-defined Data type in capnp
|
||||
* (xce_data_t tagged union) requires a proper table dispatch for non-null
|
||||
* payloads. This is a known limitation shared with wal_catchup_entry_t
|
||||
* payload; both will be resolved when the capnp schema is updated to use
|
||||
* the built-in capnp Data (List(UInt8)) type. */
|
||||
/* Chunk the snapshot file and enqueue each chunk with its file bytes.
|
||||
* Payload travels as capnp Data via xce_snapshot_chunk_req_raw (task 7.2)
|
||||
* — the generated encode path cannot carry raw bytes (xce_data_t
|
||||
* limitation, same as replConfigPush). */
|
||||
while (offset < total_size) {
|
||||
size_t chunk_len = (total_size - offset > SNAPSHOT_CHUNK_SIZE)
|
||||
? SNAPSHOT_CHUNK_SIZE
|
||||
: (size_t)(total_size - offset);
|
||||
/* Consume the file bytes to advance offset — data not used yet */
|
||||
if (fread(buf, 1, chunk_len, f) != chunk_len) {
|
||||
rc = -EIO;
|
||||
break;
|
||||
|
|
@ -1630,26 +1643,26 @@ int ha_send_snapshot_to_standby(endpoint_t *ep, const char *snap_dir,
|
|||
|
||||
snapshot_transfer_t chunk;
|
||||
memset(&chunk, 0, sizeof(chunk));
|
||||
chunk.snapshot_lsn = checkpoint_lsn;
|
||||
chunk.chunk_index = chunk_index;
|
||||
chunk.chunk_offset = offset;
|
||||
chunk.total_size = total_size;
|
||||
chunk.is_last = is_last;
|
||||
chunk.payload = NULL; /* encoding deferred */
|
||||
|
||||
xce_request_t req;
|
||||
memset(&req, 0, sizeof(req));
|
||||
req.opcode = OPCode_snapshotTransfer;
|
||||
req.body_which = Request_body_nomsg;
|
||||
req.body.snapshottransfermsg = &chunk;
|
||||
chunk.snapshot_lsn = checkpoint_lsn;
|
||||
chunk.chunk_index = chunk_index;
|
||||
chunk.chunk_offset = offset;
|
||||
chunk.total_size = total_size;
|
||||
chunk.is_last = is_last;
|
||||
chunk.raw_payload = buf; /* borrowed for hook/encode only */
|
||||
chunk.raw_payload_len = chunk_len;
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
if (xce_snapshot_chunk_hook)
|
||||
if (xce_snapshot_chunk_hook) {
|
||||
xce_request_t req;
|
||||
memset(&req, 0, sizeof(req));
|
||||
req.opcode = OPCode_snapshotTransfer;
|
||||
req.body_which = Request_body_snapshottransfermsg;
|
||||
req.body.snapshottransfermsg = &chunk;
|
||||
xce_snapshot_chunk_hook(ep, &req);
|
||||
else
|
||||
} else
|
||||
#endif
|
||||
if (ep)
|
||||
xce_request(ep, &req);
|
||||
xce_snapshot_chunk_req_raw(ep, &chunk, buf, chunk_len);
|
||||
|
||||
offset += chunk_len;
|
||||
chunk_index++;
|
||||
|
|
|
|||
|
|
@ -540,7 +540,10 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
|
|||
ectx->votes[i].peer->hostname,
|
||||
(uint32_t)ectx->votes[i].peer->port);
|
||||
if (rc >= 0)
|
||||
TRACE("HA: repl registered standby ep[%d] (slot %d)", i, rc);
|
||||
INFO("HA: repl registered standby ep[%d] (slot %d, %s:%u)",
|
||||
i, rc,
|
||||
ectx->votes[i].peer->hostname,
|
||||
ectx->votes[i].peer->port);
|
||||
else
|
||||
WARN("HA: repl_add_standby ep[%d] failed: %d", i, rc);
|
||||
}
|
||||
|
|
@ -1489,6 +1492,38 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
|
|||
* For CTRL_INIT / CTRL_ELECTING the allowlist is tighter: only
|
||||
* OP_requestVote and OP_probeConfigTag pass through.
|
||||
*/
|
||||
/* ha_has_write_quorum — check whether the cluster has enough connected
|
||||
* standbys to achieve majority for state-changing operations.
|
||||
*
|
||||
* Quorum = ceil(n_peers / 2) + 1 (self-vote included).
|
||||
* Example: n_peers=3 → quorum=2 → need ≥1 connected standby.
|
||||
*
|
||||
* Called from ha_dispatch_gate for CTRL_PRIMARY — if the cluster loses
|
||||
* majority (standby failures), write ops are rejected with -XCE_ERR_NOT_READY
|
||||
* to preserve the majority-consistency invariant.
|
||||
*
|
||||
* Read-only queries (joblist, reslist, hostlist, etc.) remain available
|
||||
* regardless of majority status — they don't mutate state. */
|
||||
static bool ha_has_write_quorum(ha_state_t *ha)
|
||||
{
|
||||
if (!ha)
|
||||
return false;
|
||||
|
||||
/* Single-node cluster: no standbys needed — self is the majority. */
|
||||
if (ha->n_peers <= 1)
|
||||
return true;
|
||||
|
||||
/* Count connected standbys. standby_endpoints is populated by
|
||||
* raft_cb_become_leader (promoted vote eps) and on_standby_connected
|
||||
* (late joiners); removed by error handlers. */
|
||||
int connected = ha->standby_endpoints
|
||||
? ha->standby_endpoints->count : 0;
|
||||
|
||||
/* Need at least n_peers/2 connected standbys to achieve quorum.
|
||||
* Integer division: n_peers/2 for n_peers=3 → 1 standby minimum. */
|
||||
return connected >= (int)(ha->n_peers / 2);
|
||||
}
|
||||
|
||||
int ha_dispatch_gate(ha_state_t *ha, int opcode)
|
||||
{
|
||||
ctrl_role_t role =
|
||||
|
|
@ -1496,6 +1531,31 @@ int ha_dispatch_gate(ha_state_t *ha, int opcode)
|
|||
|
||||
switch (role) {
|
||||
case CTRL_PRIMARY:
|
||||
/* State-changing operations require majority so committed entries
|
||||
* are durable across failover. Read-only queries and replication
|
||||
* ops are always allowed. */
|
||||
if (!ha_has_write_quorum(ha)) {
|
||||
switch (opcode) {
|
||||
/* Replication ops — always allowed (they build majority) */
|
||||
case OPCode_heartbeat:
|
||||
case OPCode_replConfig:
|
||||
case OPCode_replConfigPush:
|
||||
case OPCode_upgradeConfigAck:
|
||||
case OPCode_hello:
|
||||
case OPCode_walAck:
|
||||
case OPCode_replEntry:
|
||||
case OPCode_replAck:
|
||||
case OPCode_replCatchUpDone:
|
||||
case OPCode_walCatchupEntry:
|
||||
case OPCode_snapshotTransfer:
|
||||
/* Read-only queries */
|
||||
case OPCode_get:
|
||||
case OPCode_probeConfigTag:
|
||||
return 0;
|
||||
default:
|
||||
return -XCE_ERR_NOT_READY;
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
|
||||
case CTRL_INIT:
|
||||
|
|
@ -2623,12 +2683,18 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx)
|
|||
int peer_idx = vctx->peer_idx;
|
||||
uint64_t committed_lsn = 0;
|
||||
uint64_t committed_tag = 0;
|
||||
uint64_t last_written = 0; /* latest WAL LSN — may be > committed */
|
||||
#ifndef XCE_TESTING
|
||||
if (c) {
|
||||
committed_lsn = atomic_load_explicit(&c->persist.committed_lsn,
|
||||
memory_order_acquire);
|
||||
committed_tag = atomic_load_explicit(&ha->committed_config_tag,
|
||||
memory_order_acquire);
|
||||
/* current_lsn is the next LSN to write; last written = current - 1.
|
||||
* Read without lock — monotonic, approximate value is sufficient
|
||||
* for the replication decision. */
|
||||
last_written = (c->persist.current_lsn > 0)
|
||||
? c->persist.current_lsn - 1 : 0;
|
||||
}
|
||||
#else
|
||||
/* In test mode, c may be NULL; caller sets ha fields directly */
|
||||
|
|
@ -2658,8 +2724,15 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx)
|
|||
oldest_wal_lsn = ha_compute_oldest_wal_lsn(c->persist.wal_path);
|
||||
}
|
||||
#endif
|
||||
/* Replication target: use max(committed_lsn, last_written) so
|
||||
* standbys can catch up even when committed_lsn is stuck at 0
|
||||
* (no replicator-registered standbys to advance it). */
|
||||
uint64_t target_lsn = committed_lsn;
|
||||
if (last_written > target_lsn)
|
||||
target_lsn = last_written;
|
||||
|
||||
ha_dispatch_repl_action_for_standby(ha, c, ep, peer_idx,
|
||||
committed_tag, committed_lsn,
|
||||
committed_tag, target_lsn,
|
||||
oldest_wal_lsn);
|
||||
} else if (rep_s.data_which == RequestReply_data_walackreplymsg) {
|
||||
/* Task 4.1: WAL ACK reply from standby — extract lsn + success,
|
||||
|
|
@ -3750,8 +3823,8 @@ static void on_standby_connected(endpoint_t *ep)
|
|||
int rc = repl_add_standby(&c->persist.repl, ep,
|
||||
peer->hostname, (uint32_t)peer->port);
|
||||
if (rc >= 0)
|
||||
TRACE("HA: repl registered late-joiner standby peer[%d] (slot %d)",
|
||||
sctx->peer_idx, rc);
|
||||
INFO("HA: repl registered late-joiner standby peer[%d] (slot %d, %s:%u)",
|
||||
sctx->peer_idx, rc, peer->hostname, (uint32_t)peer->port);
|
||||
else
|
||||
WARN("HA: repl_add_standby late-joiner peer[%d] failed: %d",
|
||||
sctx->peer_idx, rc);
|
||||
|
|
|
|||
|
|
@ -525,16 +525,12 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
|
|||
return -errno;
|
||||
}
|
||||
|
||||
reply.status = 0;
|
||||
reply.data_which = RequestReply_data_jobid;
|
||||
reply.data.jobid = jobid;
|
||||
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
|
||||
/* Build job list for the AFTER_EVENT hook BEFORE replying to client.
|
||||
* The AFTER_EVENT hook writes WAL and waits for majority replication —
|
||||
* the client must not be acknowledged until the job is durable. */
|
||||
xlist_init(&jlist, submit->array_size);
|
||||
|
||||
for (i = 0; i < submit->array_size; i++) {
|
||||
/* Run hooks to tell scheduler */
|
||||
lock(c->jobmgr, RTTI_LOCK_READ);
|
||||
rc = key_find(c->jobmgr, (void*)(jobid + i), (void**)&j);
|
||||
if (j != NULL) {
|
||||
|
|
@ -554,13 +550,33 @@ int handle_jobsubmit(cluster_t* c, endpoint_t* ep, xce_request_t* request)
|
|||
}
|
||||
}
|
||||
|
||||
xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB, HOOK_AFTER_EVENT,
|
||||
HOOK_DETAIL_JOB_NEW, c, &jlist);
|
||||
int hook_rc = xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_JOB,
|
||||
HOOK_AFTER_EVENT,
|
||||
HOOK_DETAIL_JOB_NEW, c, &jlist);
|
||||
|
||||
xlist_iter(&jlist, (xlist_iter_fn_t)xlist_deref_job_item_, NULL);
|
||||
|
||||
xlist_fini(&jlist);
|
||||
|
||||
/* If persistence failed (replication timeout, I/O error), reject the
|
||||
* submission so the client knows the job was not made durable. */
|
||||
if (hook_rc != 0) {
|
||||
WARN("handle_jobsubmit: persistence hook failed (rc=%d) — rejecting job",
|
||||
hook_rc);
|
||||
reply.status = hook_rc;
|
||||
reply.data_which = RequestReply_data_nodata;
|
||||
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
return hook_rc;
|
||||
}
|
||||
|
||||
/* Reply only AFTER WAL write + majority replication succeeded. */
|
||||
reply.status = 0;
|
||||
reply.data_which = RequestReply_data_jobid;
|
||||
reply.data.jobid = jobid;
|
||||
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1038,7 +1038,14 @@ int persist_wal_sync_append(persist_t* p, uint32_t type,
|
|||
}
|
||||
if (sync_rc_out)
|
||||
*sync_rc_out = sync_err;
|
||||
/* sync_err < 0 = degraded mode (local-only durability); continue operating. */
|
||||
/* sync_err < 0 — replication failed.
|
||||
* -ENODEV: no standbys registered (single-node or degraded).
|
||||
* -ETIMEDOUT: standbys were registered but didn't ACK in time.
|
||||
* Only -ENODEV is acceptable for forward progress; timeout means
|
||||
* the cluster had standbys but couldn't reach quorum — the write
|
||||
* must be rejected to preserve the majority-consistency invariant. */
|
||||
if (sync_err == -ETIMEDOUT)
|
||||
return -ETIMEDOUT;
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -219,6 +219,14 @@ int repl_broadcast(replicator_t *r, const wal_entry_hdr_t *hdr,
|
|||
|
||||
mylock_lock(&r->lock);
|
||||
|
||||
if (r->nstandbys == 0) {
|
||||
WARN("repl_broadcast: no standbys registered — WAL entry lsn=%lu type=%u"
|
||||
" not replicated",
|
||||
(unsigned long)hdr->lsn, hdr->type);
|
||||
mylock_unlock(&r->lock);
|
||||
return 0;
|
||||
}
|
||||
|
||||
for (int i = 0; i < MAX_STANDBYS; i++) {
|
||||
standby_conn_t *s = &r->standbys[i];
|
||||
|
||||
|
|
|
|||
|
|
@ -357,6 +357,8 @@ static void test_become_follower_closes_vote_endpoints(void)
|
|||
ha_phase2_hook = NULL;
|
||||
atomic_store_explicit(&ha.elect_ctx, NULL, memory_order_release);
|
||||
free_elect_ctx(ctx);
|
||||
free(ep0);
|
||||
free(ep1);
|
||||
ha_fini(&ha);
|
||||
printf("test_become_follower_closes_vote_endpoints: PASS\n");
|
||||
}
|
||||
|
|
@ -494,6 +496,8 @@ static void test_follower_passive_no_outbound_connect(void)
|
|||
ha_phase2_hook = NULL;
|
||||
atomic_store_explicit(&ha.elect_ctx, NULL, memory_order_release);
|
||||
free_elect_ctx(ctx);
|
||||
free(ep0);
|
||||
free(ep1);
|
||||
ha_fini(&ha);
|
||||
printf("test_follower_passive_no_outbound_connect: PASS\n");
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue