feat(ha-convergence-recovery): self-restart primitive + demotion convergence (task 2.2)
This commit is contained in:
parent
c8e55947c2
commit
09103a3141
7 changed files with 337 additions and 43 deletions
|
|
@ -18,14 +18,14 @@
|
|||
- Observable completion: all task 1.1 tests PASS; grep confirms no status/API read path calls the committed-only wrapper
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4_
|
||||
|
||||
- [ ] 2. Demotion convergence: self-restart primitive and the truncate-on-demotion path
|
||||
- [x] 2. Demotion convergence: self-restart primitive and the truncate-on-demotion path
|
||||
- [x] 2.1 Write RED tests for the self-restart primitive and demotion sequence
|
||||
- Create `testsuite/test_demotion_convergence.c` and register it in meson; declare `controller_saved_argv`, `ha_self_restart`, and the `XCE_TESTING` seams (`ha_restart_hook`, `ha_fatal_hook`) in ha.h so tests compile
|
||||
- Tests: become-follower invoked while role is CTRL_PRIMARY → persistence hooks disabled (verify via `hooks_registered` reset and a hook-counter probe) BEFORE further events are persisted, `persist_truncate_wal_above` called with the `committed_lsn` cutoff, WARN with both LSNs emitted, `ha_restart_hook` invoked exactly once; `ha_self_restart` with a failing restart hook path → `ha_fatal_hook` invoked (FATAL contract); become-follower from STANDBY/ELECTING/INIT → no truncation, no restart (regression); integration: append a committed prefix + uncommitted suffix, run the demotion seam, then replay the surviving WAL into a fresh context → committed-only state (provisional entries absent)
|
||||
- Observable completion: new cases FAIL (red) — current `raft_cb_become_follower` (ha.c:627-690 anchor) neither truncates nor disables hooks
|
||||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 2.4_
|
||||
|
||||
- [ ] 2.2 Implement the self-restart primitive and demotion convergence to green
|
||||
- [x] 2.2 Implement the self-restart primitive and demotion convergence to green
|
||||
- main.c: save the NULL-terminated argv before `argp_parse` (main.c:112 anchor); expose `controller_saved_argv()`
|
||||
- ha.c: `ha_self_restart(c)` — flush logs, `execv(saved_argv[0], saved_argv)`; on exec return → ERROR + `abort()`; under XCE_TESTING invoke `ha_restart_hook` (and `ha_fatal_hook` instead of abort); audit/ensure listener and peer fds are CLOEXEC so the re-exec'd image inherits no live sockets
|
||||
- `raft_cb_become_follower`: read prior role before the role store; was-PRIMARY branch disables the three persistence hooks inline (reuse the `ha_demote_to_standby` disable sequence, ha.c:1712-1726 anchor) then schedules a one-shot delaymgr convergence task: truncate above `committed_lsn` → WARN range → `ha_self_restart` (file I/O and exec never on the callback thread)
|
||||
|
|
@ -99,4 +99,7 @@
|
|||
- **`restore_job` gained an `entry_lsn` parameter** (replay tagging). Callers updated: cluster.h decl, job.c def, persist.c extern + apply_wal_entry caller, test_repl_apply.c stub. Any new caller must pass the WAL entry header LSN (`hdr->lsn`), or `0` if unknown (0 = exportable).
|
||||
- **Build note**: the full meson controller suite SIGTERMs at the 300s wall-clock cap on the slow commit-barrier timeout suites (pre-existing). For per-task GREEN confirmation, run `controller_unit_test -t <suite>` directly with `ASAN_OPTIONS=detect_leaks=0` (harness teardown leaks are a pre-existing accepted pattern; the meson gate sets the same option).
|
||||
- **Task 2.1 RED scaffold + anchor drift (post ha-single-leader-role)**: throwaway stubs live in new `src/controller/ha_convergence.c` (`controller_saved_argv`→NULL, `ha_self_restart`→return 0, seam pointers `ha_restart_hook`/`ha_fatal_hook`=NULL) — **task 2.2 moves the real `controller_saved_argv` to main.c and `ha_self_restart` + the demotion branch to ha.c** per design file-layout. A test-only seam `raft_cb_become_follower_with_cluster_for_test` (preserves `raft_c=c`) was added to ha.c. Verified anchors: `raft_cb_become_follower` ha.c:622; `ha_demote_to_standby` ha.c:1711 (hook-disable sequence to reuse); `persist_truncate_wal_above` persist.c:1545; role enum CTRL_PRIMARY=0/STANDBY=1/INIT=2/ELECTING=3 (ha.h:59-64); `committed_lsn`/`current_lsn` on `persist_t` in src/include/persist.h; MAX_STANDBYS=3.
|
||||
- **Follow-up for task 2.2**: add a `CTRL_INIT` non-PRIMARY regression case to `test_demotion_convergence.c` (currently only STANDBY+ELECTING are exercised though the comment says "ELECTING/INIT") and correct/keep the comment accurate. 1.5 (WARN with both LSNs) is verified only by the truncation-cutoff proxy — no log-capture harness exists; covered directly only at E2E (task 8).
|
||||
- **Follow-up for task 2.2**: add a `CTRL_INIT` non-PRIMARY regression case to `test_demotion_convergence.c` (currently only STANDBY+ELECTING are exercised though the comment says "ELECTING/INIT") and correct/keep the comment accurate. 1.5 (WARN with both LSNs) is verified only by the truncation-cutoff proxy — no log-capture harness exists; covered directly only at E2E (task 8). [DONE in 2.2: INIT case added, comment corrected.]
|
||||
- **Task 2.2 — latent persist bug fixed**: `persist_truncate_wal_above` (persist.c) had a no-op early-return for `cutoff_seg_idx < 0` (cutoff strictly below the oldest retained segment — the first real caller is the demotion path, where a demoted ex-primary's committed boundary can sit below its entire retained log). Now deletes all segments, sets `current_lsn = cutoff+1`, `committed_lsn = cutoff` — correct truncate-above semantics; reviewed safe for the checkpoint path (P0 `load_snapshot` restores only LSN counters, no entity state; entity-bearing snapshots out of scope) and the promotion-truncation caller (no data loss; the discriminant cannot conflate "below oldest" with "above newest"). **Follow-up (non-blocking, for task 6.1 or 7): add a targeted `persist_truncate_wal_above` test exercising the `cutoff_seg_idx < 0` branch** — currently established correct by analysis only, no direct assertion.
|
||||
- **CLOEXEC**: controller sockets are already `FD_CLOEXEC` at creation (`socket_create` netio.c:219, `socket_accept` netio.c:308) — the re-exec'd image inherits no live sockets; no netio.c change was needed.
|
||||
- **Disk space**: heavy nix builds can fill the root overlay with stale `/tmp/nix-shell.*` dirs (a reviewer hit "No space left on device" and reclaimed ~16GB). If a build fails on disk space, clear stale nix-shell temp dirs and retry — not a code failure.
|
||||
|
|
|
|||
|
|
@ -662,11 +662,53 @@ static void raft_cb_become_follower(void *ctx, uint64_t term, int leader_idx)
|
|||
}
|
||||
}
|
||||
|
||||
/* ha-convergence-recovery Req 1.1-1.3, 1.5: demotion convergence.
|
||||
*
|
||||
* Read the PRIOR role BEFORE the CTRL_STANDBY store below. If this node was
|
||||
* the PRIMARY, it may hold an uncommitted, potentially divergent WAL suffix
|
||||
* and provisional in-memory state. Synchronously disable the three
|
||||
* persistence hooks HERE (before any further event can be persisted — Req
|
||||
* 1.1, reusing ha_demote_to_standby's disable sequence), then schedule a
|
||||
* one-shot delaymgr task that truncates above committed_lsn and self-restarts
|
||||
* (Req 1.2/1.3/1.5) off this callback thread (D1: no file I/O or exec on the
|
||||
* iomgr/raft callback thread).
|
||||
*
|
||||
* Non-PRIMARY demotions (STANDBY/INIT/ELECTING) take the unchanged path —
|
||||
* they have no registered hooks and no provisional suffix to discard, so
|
||||
* their behavior is byte-equivalent to before this spec. */
|
||||
ctrl_role_t prior_role = (ctrl_role_t)atomic_load_explicit(
|
||||
&ha->role, memory_order_acquire);
|
||||
|
||||
if (prior_role == CTRL_PRIMARY && c != NULL) {
|
||||
/* Req 1.1: disable persistence hooks SYNCHRONOUSLY, before the role
|
||||
* store and before scheduling, so no provisional entry can be produced
|
||||
* after the higher term was observed. Mirrors ha_demote_to_standby
|
||||
* (ha.c) — disable each hook, then clear the idempotency guard. */
|
||||
if (ha->hooks_registered) {
|
||||
xce_disable_hook(&c->hooksys, XCE_CONTROLLER_HOOK_JOB,
|
||||
(hook_fn_t)persist_job_hook_fun);
|
||||
xce_disable_hook(&c->hooksys, XCE_CONTROLLER_HOOK_QUEUE,
|
||||
(hook_fn_t)persist_queue_hook_fun);
|
||||
xce_disable_hook(&c->hooksys, XCE_CONTROLLER_HOOK_WORKER,
|
||||
(hook_fn_t)persist_worker_hook_fun);
|
||||
ha->hooks_registered = 0;
|
||||
}
|
||||
}
|
||||
|
||||
atomic_store_explicit(&ha->role, (int32_t)CTRL_STANDBY,
|
||||
memory_order_release);
|
||||
INFO("HA FSM (raft): → CTRL_STANDBY (term=%" PRIu64 ", first_boot=%d)",
|
||||
term, is_first_boot);
|
||||
|
||||
/* Req 1.2/1.3/1.5: schedule the deferred truncate-above-committed-LSN +
|
||||
* self-restart now that hooks are disabled and the role is STANDBY. Only
|
||||
* the was-PRIMARY edge converges; all other edges skip this entirely. */
|
||||
if (prior_role == CTRL_PRIMARY && c != NULL) {
|
||||
WARN("HA: PRIMARY demoted (term=%" PRIu64 ") — scheduling demotion "
|
||||
"convergence (truncate above committed_lsn + self-restart)", term);
|
||||
(void)ha_schedule_demotion_convergence(c);
|
||||
}
|
||||
|
||||
/* Arm heartbeat watchdog — fires ha_on_primary_ctrl_fd_rdhup if primary
|
||||
* sends no ctrlhello within election_timeout_ms. */
|
||||
ha_arm_heartbeat_timeout_vtask(ha, c);
|
||||
|
|
|
|||
|
|
@ -273,6 +273,26 @@ char **controller_saved_argv(void);
|
|||
*/
|
||||
int ha_self_restart(cluster_t *c);
|
||||
|
||||
/*
|
||||
* ha_schedule_demotion_convergence: schedule the one-shot delaymgr task that
|
||||
* truncates above committed_lsn (Req 1.2), WARN-logs the discarded range
|
||||
* (Req 1.5), and self-restarts (Req 1.3). Called from raft_cb_become_follower's
|
||||
* was-PRIMARY branch AFTER the persistence hooks are disabled inline (Req 1.1),
|
||||
* so all file I/O and the exec run off the become-follower callback thread (D1).
|
||||
* Returns 0 on success, negative on alloc/schedule failure (suffix is then
|
||||
* reconciled by rejoin reconciliation at the next join).
|
||||
*/
|
||||
int ha_schedule_demotion_convergence(cluster_t *c);
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
/*
|
||||
* ha_demotion_convergence_fn: the delaymgr task body — exposed under XCE_TESTING
|
||||
* only so the demotion-convergence suite can reference its contract; the test
|
||||
* fixture drives the queued task through its stored entry pointer.
|
||||
*/
|
||||
int ha_demotion_convergence_fn(void *arg);
|
||||
#endif
|
||||
|
||||
|
||||
/* ── Redirect helpers used by join.c ──────────────────────── */
|
||||
|
||||
|
|
|
|||
|
|
@ -3,34 +3,52 @@
|
|||
* (ha-convergence-recovery Requirement 1).
|
||||
*
|
||||
* ───────────────────────────────────────────────────────────────────────────
|
||||
* RED SCAFFOLD (task 2.1) — task 2.2 REPLACES the bodies below with the real
|
||||
* implementation (and moves controller_saved_argv's backing store into main.c).
|
||||
* This TU owns the self-restart primitive and the deferred demotion-convergence
|
||||
* task body. The synchronous demotion BRANCH (hook disable + task scheduling)
|
||||
* lives in raft_cb_become_follower (ha.c) because that callback is file-static;
|
||||
* it calls ha_schedule_demotion_convergence() declared here.
|
||||
*
|
||||
* This file currently provides ONLY the link-time definitions the task-2.1 RED
|
||||
* tests need to compile and link:
|
||||
* - controller_saved_argv() — stub returning NULL (real argv capture is in
|
||||
* main.c, task 2.2)
|
||||
* - ha_self_restart() — RED stub: does NOT yet honor the ha_restart_hook
|
||||
* / ha_fatal_hook contract, so the fatal-hook test
|
||||
* FAILS until task 2.2 implements the real
|
||||
* flush → execv → abort sequence (and the
|
||||
* XCE_TESTING seam routing).
|
||||
* - ha_restart_hook / ha_fatal_hook — the XCE_TESTING function-pointer seams,
|
||||
* defined =NULL in exactly one TU (this one),
|
||||
* mirroring the ha_*_hook seam pattern in ha.c.
|
||||
* Split of responsibilities (design "Demotion Convergence + Self-Restart"):
|
||||
* - raft_cb_become_follower (ha.c): read prior role → if was-PRIMARY, disable
|
||||
* the 3 persistence hooks INLINE (Req 1.1),
|
||||
* then schedule the one-shot convergence
|
||||
* task off the callback thread (D1).
|
||||
* - ha_demotion_convergence_fn (here, delaymgr task body): truncate above
|
||||
* committed_lsn (Req 1.2) → WARN range
|
||||
* (Req 1.5) → ha_self_restart (Req 1.3).
|
||||
* - ha_self_restart (here): flush logs → execv(saved argv); exec
|
||||
* return → ERROR + abort (FATAL, Req 1.4);
|
||||
* under XCE_TESTING route to the restart /
|
||||
* fatal seams (Req 1.6, 1.4).
|
||||
*
|
||||
* The DEMOTION-SEQUENCE behavior (hook disable on become-follower-from-PRIMARY,
|
||||
* scheduled truncate-above-committed_lsn, restart) lives in raft_cb_become_
|
||||
* follower (ha.c) and is NOT implemented here or there yet — that is task 2.2.
|
||||
* Consequently every demotion-convergence test in test_demotion_convergence.c
|
||||
* is RED against the current tree.
|
||||
* controller_saved_argv lives in main.c (the real argv capture); this TU only
|
||||
* consumes it through the ha.h accessor.
|
||||
*
|
||||
* fd hygiene (Req 1.3 / design fd audit): every controller-to-controller socket
|
||||
* — listener, accepted peer/replication connections, and outbound vote/peer
|
||||
* connect sockets — is created FD_CLOEXEC in src/lib/netio.c (socket_create
|
||||
* sets it at netio.c:219; socket_accept at netio.c:308). The re-exec'd image
|
||||
* therefore inherits no live sockets and re-creates all endpoints through the
|
||||
* boot path; no extra close-before-exec sweep is required here.
|
||||
* ───────────────────────────────────────────────────────────────────────────
|
||||
*/
|
||||
|
||||
#define _GNU_SOURCE
|
||||
#include <errno.h>
|
||||
#include <inttypes.h>
|
||||
#include <pthread.h>
|
||||
#include <stdatomic.h>
|
||||
#include <stddef.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <tlog.h>
|
||||
|
||||
#include "cluster.h"
|
||||
#include "delay.h"
|
||||
#include "ha.h"
|
||||
#include "persist.h"
|
||||
#include "xce/cluster_def.h"
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
|
|
@ -41,30 +59,154 @@ void (*ha_fatal_hook)(const char *msg) = NULL;
|
|||
#endif
|
||||
|
||||
/*
|
||||
* controller_saved_argv — RED stub (task 2.2 moves the real, main()-populated
|
||||
* accessor into main.c). Returns NULL: no argv has been captured in the
|
||||
* scaffold, so a production execv would have nothing to exec.
|
||||
*/
|
||||
char **controller_saved_argv(void)
|
||||
{
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* ha_self_restart — RED stub.
|
||||
* ha_self_restart — the truncate-and-restart primitive's restart half.
|
||||
*
|
||||
* Task 2.2 replaces this with the real primitive:
|
||||
* - production: flush logs, execv(controller_saved_argv()); on exec return,
|
||||
* ERROR + abort() (FATAL — Req 1.4); never returns.
|
||||
* - XCE_TESTING: invoke ha_restart_hook(c) instead of execv (Req 1.6); if it
|
||||
* reports failure (or is unset), invoke ha_fatal_hook in place of abort.
|
||||
* Production (Req 1.3, 1.4): flush logs, then execv() the saved argv so the
|
||||
* controller rebuilds in-memory state purely from the surviving (committed)
|
||||
* WAL prefix via boot recovery. If exec cannot be performed, terminate
|
||||
* FATALLY (abort) — a demoted ex-primary must NEVER continue running with
|
||||
* potentially divergent state. NEVER returns in production.
|
||||
*
|
||||
* Until then this stub deliberately does NOTHING (no hook routing), so the
|
||||
* task-2.1 fatal-hook test — which asserts a failing restart hook drives
|
||||
* ha_fatal_hook — FAILS (RED).
|
||||
* Test (Req 1.6): under XCE_TESTING invoke ha_restart_hook(c) instead of
|
||||
* execv and return its result on the success path; a non-zero hook return
|
||||
* models an exec failure and routes to ha_fatal_hook in place of abort, so a
|
||||
* test binary can assert the FATAL path was taken without dying (Req 1.4).
|
||||
*/
|
||||
int ha_self_restart(cluster_t *c)
|
||||
{
|
||||
#ifdef XCE_TESTING
|
||||
/* Test contract: the restart seam stands in for execv. */
|
||||
int rc = 0;
|
||||
if (ha_restart_hook != NULL)
|
||||
rc = ha_restart_hook(c);
|
||||
|
||||
if (rc != 0 || ha_restart_hook == NULL) {
|
||||
/* Restart could not be performed → FATAL contract (Req 1.4). */
|
||||
if (ha_fatal_hook != NULL)
|
||||
ha_fatal_hook("ha_self_restart: restart seam failed (FATAL)");
|
||||
return rc != 0 ? rc : -1;
|
||||
}
|
||||
return 0;
|
||||
#else
|
||||
(void)c;
|
||||
return 0; /* RED scaffold — task 2.2 implements exec/abort + seam routing */
|
||||
|
||||
char **argv = controller_saved_argv();
|
||||
if (argv == NULL || argv[0] == NULL) {
|
||||
ERROR("HA self-restart: no saved argv — cannot re-exec (FATAL)");
|
||||
tlog_exit(); /* flush pending log buffer before dying */
|
||||
abort();
|
||||
}
|
||||
|
||||
INFO("HA self-restart: re-executing %s to rebuild from committed state",
|
||||
argv[0]);
|
||||
tlog_exit(); /* flush + drain the async log buffer before exec replaces us */
|
||||
|
||||
execv(argv[0], argv);
|
||||
|
||||
/* execv only returns on failure — FATAL (Req 1.4): never continue serving
|
||||
* as a divergent ex-primary. */
|
||||
ERROR("HA self-restart: execv(%s) failed: %d — aborting (FATAL)",
|
||||
argv[0], errno);
|
||||
abort();
|
||||
return -1; /* unreachable */
|
||||
#endif
|
||||
}
|
||||
|
||||
/*
|
||||
* ha_demotion_convergence_ctx_t — heap context for the one-shot delaymgr task.
|
||||
* Holds only the cluster pointer; committed_lsn is read at task time (not
|
||||
* schedule time) so a late-advancing commit still truncates to the right
|
||||
* boundary.
|
||||
*/
|
||||
typedef struct {
|
||||
cluster_t *c;
|
||||
} ha_demotion_convergence_ctx_t;
|
||||
|
||||
/*
|
||||
* ha_demotion_convergence_fn — delaymgr task body (runs OFF the become-follower
|
||||
* callback thread, D1). Truncate above committed_lsn (Req 1.2), WARN with the
|
||||
* truncated range (Req 1.5), then self-restart (Req 1.3). On truncation failure
|
||||
* (Req error strategy) ERROR then restart anyway — the rejoin reconciliation
|
||||
* loop converges the untruncated case at the next join.
|
||||
*
|
||||
* In XCE_TESTING the test fixture drives this entry directly (un-started
|
||||
* delaymgr); ha_self_restart routes to the restart seam there. The ctx is
|
||||
* freed before restart in production restart never returns, so only the test
|
||||
* path reaches the free — kept on every non-restarting exit for safety.
|
||||
*/
|
||||
int ha_demotion_convergence_fn(void *arg)
|
||||
{
|
||||
ha_demotion_convergence_ctx_t *ctx = (ha_demotion_convergence_ctx_t *)arg;
|
||||
if (ctx == NULL || ctx->c == NULL) {
|
||||
free(ctx);
|
||||
return -EINVAL;
|
||||
}
|
||||
cluster_t *c = ctx->c;
|
||||
|
||||
uint64_t committed = atomic_load_explicit(&c->persist.committed_lsn,
|
||||
memory_order_acquire);
|
||||
|
||||
/* Prior log tip for the WARN range (read under the persist lock). */
|
||||
pthread_mutex_lock(&c->persist.lock);
|
||||
uint64_t prior_tip = c->persist.current_lsn > 0
|
||||
? c->persist.current_lsn - 1
|
||||
: 0;
|
||||
pthread_mutex_unlock(&c->persist.lock);
|
||||
|
||||
int trc = persist_truncate_wal_above(&c->persist, committed);
|
||||
if (trc < 0) {
|
||||
ERROR("HA demotion convergence: truncate above committed_lsn=%" PRIu64
|
||||
" failed: %d — restarting anyway (rejoin reconciliation will "
|
||||
"converge)", committed, trc);
|
||||
} else {
|
||||
/* Req 1.5: log the discarded range — (committed_lsn, prior tip]. */
|
||||
WARN("HA demotion convergence: truncated WAL above committed_lsn=%"
|
||||
PRIu64 " (prior tip=%" PRIu64 "); discarding provisional suffix, "
|
||||
"restarting to rebuild from committed state",
|
||||
committed, prior_tip);
|
||||
}
|
||||
|
||||
/* Free the task ctx before restart (production exec never returns). */
|
||||
free(ctx);
|
||||
|
||||
/* Req 1.3: rebuild in-memory state through boot recovery. */
|
||||
return ha_self_restart(c);
|
||||
}
|
||||
|
||||
/*
|
||||
* ha_schedule_demotion_convergence — called from raft_cb_become_follower's
|
||||
* was-PRIMARY branch AFTER the persistence hooks have been disabled inline.
|
||||
* Schedules exactly one one-shot delaymgr task running ha_demotion_convergence_fn
|
||||
* so the file I/O (truncation) and exec never run on the callback thread (D1).
|
||||
*
|
||||
* Returns 0 on success, negative on allocation/scheduling failure. On failure
|
||||
* the node stays demoted with hooks disabled; the untruncated suffix is then
|
||||
* reconciled by rejoin reconciliation at the next join (crash-safe by design).
|
||||
*/
|
||||
int ha_schedule_demotion_convergence(cluster_t *c)
|
||||
{
|
||||
if (c == NULL)
|
||||
return -EINVAL;
|
||||
|
||||
ha_demotion_convergence_ctx_t *ctx = calloc(1, sizeof(*ctx));
|
||||
if (ctx == NULL) {
|
||||
ERROR("HA demotion convergence: ctx alloc failed — suffix will be "
|
||||
"reconciled at next rejoin");
|
||||
return -ENOMEM;
|
||||
}
|
||||
ctx->c = c;
|
||||
|
||||
/* One-shot (runcount=1), minimal delay: the work must run promptly but off
|
||||
* the iomgr callback thread. delay_task_ms only enqueues (ordq_put); the
|
||||
* delaymgr worker runs it. In XCE_TESTING the fixture drives the queued
|
||||
* task synchronously via its stored entry/args. */
|
||||
int rc = delay_task_ms(&c->delaymgr, 1, 1,
|
||||
(delay_task_entry_fn_t)ha_demotion_convergence_fn,
|
||||
ctx);
|
||||
if (rc < 0) {
|
||||
ERROR("HA demotion convergence: delay_task_ms failed: %d", rc);
|
||||
free(ctx);
|
||||
return rc;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -109,6 +109,28 @@ static error_t parse_opt(int key, char* arg, struct argp_state* state)
|
|||
|
||||
static struct argp argp = { options, parse_opt, args_doc, doc };
|
||||
|
||||
/*
|
||||
* Saved argv for self-restart (ha-convergence-recovery Req 1.3).
|
||||
*
|
||||
* main() captures the original, NULL-terminated argv pointer before argp_parse
|
||||
* mutates the program state, so ha_self_restart can execv() the controller back
|
||||
* into existence after a demotion/rejoin convergence truncates the divergent WAL
|
||||
* suffix. argv supplied to main() is already NULL-terminated per C11 §5.1.2.2.1
|
||||
* (argv[argc] == NULL) and lives for the whole program lifetime, so storing the
|
||||
* pointer is sufficient — no copy needed.
|
||||
*/
|
||||
static char **g_saved_argv = NULL;
|
||||
|
||||
/*
|
||||
* controller_saved_argv — accessor declared in ha.h, consumed by ha_self_restart.
|
||||
* Returns the NULL-terminated argv captured by main(), or NULL if never saved
|
||||
* (e.g. unit-test binaries that do not run main()).
|
||||
*/
|
||||
char **controller_saved_argv(void)
|
||||
{
|
||||
return g_saved_argv;
|
||||
}
|
||||
|
||||
int main(int argc, char* argv[])
|
||||
{
|
||||
struct arguments arguments;
|
||||
|
|
@ -121,6 +143,10 @@ int main(int argc, char* argv[])
|
|||
char cert_file[XCE_MAXPATH_LEN + 1];
|
||||
char myname[XCE_MAXNAME_LEN + 1];
|
||||
|
||||
/* Capture the NULL-terminated argv for self-restart BEFORE argp_parse so a
|
||||
* later demotion/rejoin convergence can execv() the same image (Req 1.3). */
|
||||
g_saved_argv = argv;
|
||||
|
||||
memset(&arguments, 0x0, sizeof(arguments));
|
||||
arguments.conf_keep_n = 2;
|
||||
arguments.conf_stale_timeout = 86400;
|
||||
|
|
|
|||
|
|
@ -1597,6 +1597,30 @@ int persist_truncate_wal_above(persist_t* p, uint64_t cutoff_lsn)
|
|||
}
|
||||
|
||||
if (cutoff_seg_idx < 0) {
|
||||
/* cutoff_lsn is below the oldest retained segment's start LSN, so EVERY
|
||||
* on-disk entry has LSN > cutoff_lsn and must be removed (the function's
|
||||
* contract: "remove all entries with LSN > cutoff_lsn"). This is the
|
||||
* demotion-convergence case (ha-convergence-recovery Req 1.2): a demoted
|
||||
* ex-primary whose committed boundary sits below its entire retained log
|
||||
* must discard all of it and collapse the tip to cutoff_lsn + 1. Delete
|
||||
* every segment, close the active fd, and reset the in-memory tip. */
|
||||
for (int i = 0; i < nseg; i++) {
|
||||
char del_path[4096];
|
||||
snprintf(del_path, sizeof(del_path),
|
||||
WAL_SEG_FMT, p->wal_path, seg_lsns[i]);
|
||||
unlink(del_path);
|
||||
}
|
||||
if (p->wal_fd >= 0) {
|
||||
fdatasync(p->wal_fd);
|
||||
close(p->wal_fd);
|
||||
p->wal_fd = -1;
|
||||
}
|
||||
p->seg_offset = 0;
|
||||
p->seg_path[0] = '\0';
|
||||
p->current_lsn = cutoff_lsn + 1;
|
||||
p->last_crc = WAL_CRC_NONE;
|
||||
atomic_store_explicit(&p->committed_lsn, cutoff_lsn,
|
||||
memory_order_release);
|
||||
pthread_mutex_unlock(&p->lock);
|
||||
return 0;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -369,7 +369,7 @@ static void test_demotion_from_standby_no_truncation_no_restart(void)
|
|||
printf("test_demotion_from_standby_no_truncation_no_restart: checked\n");
|
||||
}
|
||||
|
||||
/* Regression: demotion from ELECTING/INIT must NOT truncate or restart. */
|
||||
/* Regression: demotion from ELECTING must NOT truncate or restart. */
|
||||
static void test_demotion_from_electing_no_truncation_no_restart(void)
|
||||
{
|
||||
dc_setup("electing", /*n_peers=*/3, /*n_standbys=*/2,
|
||||
|
|
@ -403,6 +403,42 @@ static void test_demotion_from_electing_no_truncation_no_restart(void)
|
|||
printf("test_demotion_from_electing_no_truncation_no_restart: checked\n");
|
||||
}
|
||||
|
||||
/* Regression: demotion from INIT must NOT truncate or restart (mirror of the
|
||||
* ELECTING case — a CTRL_INIT node has no registered persistence hooks and no
|
||||
* provisional suffix, so the was-PRIMARY convergence branch must not fire). */
|
||||
static void test_demotion_from_init_no_truncation_no_restart(void)
|
||||
{
|
||||
dc_setup("init", /*n_peers=*/3, /*n_standbys=*/2,
|
||||
/*appended_lsn=*/9, /*n_vcpus=*/8);
|
||||
atomic_store(&g_dc.persist.committed_lsn, 4);
|
||||
|
||||
/* No registered hooks on a starting node; force role to INIT. */
|
||||
TEST_ASSERT_EQUAL_INT(0, ha_demote_to_standby(&g_dc)); /* clears hooks */
|
||||
atomic_store_explicit(&g_dc.ha.role, (int32_t)CTRL_INIT,
|
||||
memory_order_release);
|
||||
|
||||
ha_restart_hook = dc_restart_hook;
|
||||
ha_fatal_hook = dc_fatal_hook;
|
||||
|
||||
uint64_t tip_before = g_dc.persist.current_lsn;
|
||||
int tasks_before = dc_task_count(&g_dc.delaymgr);
|
||||
|
||||
/* Act. */
|
||||
raft_cb_become_follower_with_cluster_for_test(&g_dc, /*term=*/8);
|
||||
|
||||
/* Assert. */
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(tasks_before,
|
||||
dc_task_count(&g_dc.delaymgr),
|
||||
"demotion from INIT must schedule NO convergence task (regression)");
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(tip_before, g_dc.persist.current_lsn,
|
||||
"demotion from INIT must NOT truncate the WAL (regression)");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, atomic_load(&g_restart_calls),
|
||||
"demotion from INIT must NOT self-restart (regression)");
|
||||
|
||||
dc_teardown();
|
||||
printf("test_demotion_from_init_no_truncation_no_restart: checked\n");
|
||||
}
|
||||
|
||||
/* 1.7/2.4 (integration, in-process half): append a committed prefix + an
|
||||
* uncommitted suffix, run the demotion seam, then replay the SURVIVING WAL into
|
||||
* a FRESH persist context. The replayed state must contain the committed prefix
|
||||
|
|
@ -492,6 +528,7 @@ void test_demotion_convergence(void)
|
|||
RUN_TEST(test_self_restart_failure_invokes_fatal_hook);
|
||||
RUN_TEST(test_demotion_from_standby_no_truncation_no_restart);
|
||||
RUN_TEST(test_demotion_from_electing_no_truncation_no_restart);
|
||||
RUN_TEST(test_demotion_from_init_no_truncation_no_restart);
|
||||
RUN_TEST(test_demotion_replay_yields_committed_only);
|
||||
#else
|
||||
printf("test_demotion_convergence: SKIP (non-test build)\n");
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue