lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: --max-wal-records CLI flag + forced-snapshot soak scaffolding (raft-ha-core-safety 5.10a / ha-entity-snapshots 7.1, partial)

Browse source 
Adds --max-wal-records (mirrors the existing --wal-sync-ms pattern:
main.c argp option -> myconf_t -> persist_config_set_max_wal_records())
so a deployment can force the WAL-record-count checkpoint trigger to
fire on a job count a soak can actually reach, instead of the
production default of 10000.

Adds a new `--force-snapshot` soak mode (run-ha-soak.sh): submits
enough jobs to cross the configured threshold, kills a non-primary
node, wipes its data dir (a genuine empty rejoin), restarts it, and
asserts the cluster reconverges with zero walcompare divergence and
zero job loss. Fails loud (not a false PASS) if the restarted node's
log shows no actual snapshot install, so the round can't silently
report success without exercising the real scenario.

INVESTIGATION FINDING (not yet resolved, this commit does not close
5.10a or ha-entity-snapshots 7.1): live-tested with temporary
diagnostic logging (since reverted). schedule_checkpoint() correctly
fires once wal_record_count reaches the configured threshold
(confirmed: "wal_record_count=100 max=100" logged), but the delaymgr-
scheduled checkpoint_task_fn never executes afterward -- no checkpoint
ever completes, no snapshot file is ever written, in a live multi-node
deployment. This is a separate, deeper issue in the delaymgr scheduling
path (or its wiring for this specific call shape), not a bug in this
commit's CLI plumbing or soak scaffolding -- both are built, tested,
and confirmed working up to the point where the async checkpoint
should run. Needs its own dedicated investigation (likely src/lib
delaymgr, not persist.c) before --force-snapshot can produce a PASS.

This also has production implications beyond the test scenario: if
confirmed, WAL checkpoints may never fire automatically today, meaning
WAL segments are never trimmed via this path in a live deployment.

controller_unit_test: test_ha_startup 34/0, test_ha_core_safety 28/0,
test_persist_checkpoint 5/0, test_walcompare 17/0 (no regression).
Soak --selftest 17/17.
This commit is contained in:
Rongsong Shen 2026-07-02 03:19:40 +00:00
parent f3c86a683a
commit 9f4292de29
6 changed files with 179 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

14
src/controller/main.c
View file

@ -31,6 +31,7 @@ const char* argp_program_bug_address = BUG_EMAIL;
#define OPT_CERTDIR 260
#define OPT_SYNC_TIMEOUT 261
#define OPT_WAL_SYNC 262
#define OPT_MAX_WAL_RECORDS 263
struct arguments {
char* myname;
@ -44,6 +45,7 @@ struct arguments {
char* certdir;
uint32_t sync_timeout_ms;
uint32_t wal_sync_ms;
uint32_t max_wal_records;
};
static struct argp_option options[] = {
@ -66,6 +68,9 @@ static struct argp_option options[] = {
"Synchronous majority-replication commit barrier timeout in ms (0 = default)" },
{ "wal-sync-ms", OPT_WAL_SYNC, "MS", 0,
"WAL flush (fsync) cadence in ms (0 = default)" },
{ "max-wal-records", OPT_MAX_WAL_RECORDS, "N", 0,
"WAL records since last checkpoint before an async checkpoint is "
"scheduled (0 = default; persist enforces a minimum of 100)" },
{ 0 }
};
@ -122,6 +127,13 @@ static error_t parse_opt(int key, char* arg, struct argp_state* state)
arguments->wal_sync_ms = (uint32_t)v;
break;
}
case OPT_MAX_WAL_RECORDS: {
int v = atoi(arg);
if (v < 0)
argp_error(state, "--max-wal-records must be >= 0 (got: %s)", arg);
arguments->max_wal_records = (uint32_t)v;
break;
}
default:
return ARGP_ERR_UNKNOWN;
}
@ -175,6 +187,7 @@ int main(int argc, char* argv[])
arguments.vcpus = 0; /* 0 = not set; use cluster config default */
arguments.sync_timeout_ms = 0; /* 0 = not set; use persist config default */
arguments.wal_sync_ms = 0; /* 0 = not set; use persist config default */
arguments.max_wal_records = 0; /* 0 = not set; use persist config default */
argp_parse(&argp, argc, argv, 0, 0, &arguments);
@ -225,6 +238,7 @@ int main(int argc, char* argv[])
myconf.vcpus = arguments.vcpus;
myconf.sync_timeout_ms = arguments.sync_timeout_ms;
myconf.wal_sync_ms = arguments.wal_sync_ms;
myconf.max_wal_records = arguments.max_wal_records;
memset(&myconf.peers_cfg, 0, sizeof(myconf.peers_cfg));
if (arguments.peers_str) {