lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: --max-wal-records CLI flag + forced-snapshot soak scaffolding (raft-ha-core-safety 5.10a / ha-entity-snapshots 7.1, partial)

Browse source 
Adds --max-wal-records (mirrors the existing --wal-sync-ms pattern:
main.c argp option -> myconf_t -> persist_config_set_max_wal_records())
so a deployment can force the WAL-record-count checkpoint trigger to
fire on a job count a soak can actually reach, instead of the
production default of 10000.

Adds a new `--force-snapshot` soak mode (run-ha-soak.sh): submits
enough jobs to cross the configured threshold, kills a non-primary
node, wipes its data dir (a genuine empty rejoin), restarts it, and
asserts the cluster reconverges with zero walcompare divergence and
zero job loss. Fails loud (not a false PASS) if the restarted node's
log shows no actual snapshot install, so the round can't silently
report success without exercising the real scenario.

INVESTIGATION FINDING (not yet resolved, this commit does not close
5.10a or ha-entity-snapshots 7.1): live-tested with temporary
diagnostic logging (since reverted). schedule_checkpoint() correctly
fires once wal_record_count reaches the configured threshold
(confirmed: "wal_record_count=100 max=100" logged), but the delaymgr-
scheduled checkpoint_task_fn never executes afterward -- no checkpoint
ever completes, no snapshot file is ever written, in a live multi-node
deployment. This is a separate, deeper issue in the delaymgr scheduling
path (or its wiring for this specific call shape), not a bug in this
commit's CLI plumbing or soak scaffolding -- both are built, tested,
and confirmed working up to the point where the async checkpoint
should run. Needs its own dedicated investigation (likely src/lib
delaymgr, not persist.c) before --force-snapshot can produce a PASS.

This also has production implications beyond the test scenario: if
confirmed, WAL checkpoints may never fire automatically today, meaning
WAL segments are never trimmed via this path in a live deployment.

controller_unit_test: test_ha_startup 34/0, test_ha_core_safety 28/0,
test_persist_checkpoint 5/0, test_walcompare 17/0 (no regression).
Soak --selftest 17/17.
This commit is contained in:
Rongsong Shen 2026-07-02 03:19:40 +00:00
parent f3c86a683a
commit 9f4292de29
6 changed files with 179 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

6
src/controller/cluster.c
View file

@ -1324,6 +1324,12 @@ int cluster_init_phase2(cluster_t* c)
* the requested cadence (ha-commit-durability Req 3.1). */
persist_config_set_wal_sync_ms(&pcfg, c->myconf->wal_sync_ms);
/* --max-wal-records checkpoint-trigger override (0 = keep the
* persist default). raft-ha-core-safety task 5.10a / ha-entity-
* snapshots task 7.1: lets a soak force real checkpoints on a
* deterministic record count instead of the 10000-record default. */
persist_config_set_max_wal_records(&pcfg, c->myconf->max_wal_records);
persist_set_workdir(&c->persist, workdir, name);
rc = persist_init(&c->persist, &pcfg, c);
if (rc != 0) {