lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(config-upgrade-snapshot): live config-upgrade soak infra + fix bug-1468/bug-1469 (task 8.6)

Browse source 
Build the live 3-node chaotic-multi-kill config-upgrade soak scenario
(run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1,
6.4. This needed genuinely new client-side infrastructure: a hand-rolled
capnp send path for OP_upgradeConfig (the auto-generated encoder is a
no-op for its Data fields, worked around the same way
xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig
CLI commands.

Building and running this live surfaced two real production bugs, both
fixed here:

- bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's
  primary/standby dispatch guard in cluster.c, making config upgrades
  unreachable from any network client.
- bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called
  config_activate_committed(), so the node that originates a config
  upgrade never refreshed its own in-memory state as part of committing
  it. Killing that node before some later, unrelated event happened to
  refresh it left its checkpoint permanently stale -- the first live,
  empirical reproduction of bug-857's originally hypothesized failure
  mode. Fixed by activating before the marker-triggered checkpoint write,
  since checkpoint content is built from live runtime tables that only
  reflect the reload after activation runs.

A separate, narrower checkpoint divergence (bug-1471, not config-related)
remains open and still blocks a clean soak PASS -- task 8.6 stays
in-progress until it's resolved.
This commit is contained in:
Rongsong Shen 2026-07-25 15:48:20 +00:00
parent 2bd44e8e4d
commit 959b0cdacb
13 changed files with 1022 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

110
src/controller/persist.c
View file

@ -56,6 +56,20 @@ extern int restore_host_state(cluster_t* c, const char* name,
* initialized default. */
extern int upgrade_materialization_verify(cluster_t* c);
/* config_activate_committed defined in config_repl.c (config-upgrade-snapshot
* task 5/6.2 remediation, bug-1469, Req 1.8/3.2/3.4) -- re-parses the
* locally-held config bytes referenced by the latest committed
* WAL_CONFIG_MARKER into c->myconf (load_primary_config), hash-verifying
* them first. Called unconditionally from apply_committed_entry's
* WAL_CONFIG_MARKER case below, BEFORE upgrade_materialization_verify: every
* member's own live apply of a marker it just committed must refresh its
* own c->myconf, not only members that separately receive an out-of-band
* handle_repl_config_push (see the call site's own doc comment for the full
* rationale). Declared here via extern, mirroring upgrade_materialization_
* verify just above, rather than pulling in config_repl.h's full include
* surface for one symbol. */
extern int config_activate_committed(cluster_t* c);
/* raft-apply-decoupling task 3.2: pending command-apply hand-off list node
* (producer side; the not-yet-built ordered apply pump, task 6.1/6.2, is
* the consumer). Declared here (rather than persist.h) so the list stays
@ -3807,6 +3821,102 @@ int apply_committed_entry(persist_t* p, uint32_t wal_type, uint64_t lsn,
* checkpoint. `lsn` (this function's own parameter) is
* already the marker's exact position, so it is passed
* through explicitly instead of re-derived. */
/* bug-1469 fix (live HA soak, config-upgrade-snapshot task 8.6):
* unconditionally re-run config activation right here, BEFORE the
* forced checkpoint below, so EVERY member -- primary, standby, and
* any future newly-elected leader -- is guaranteed to have
* refreshed c->myconf (and the runtime queue/host tables
* load_primary_config provisions from it) from disk BEFORE the
* checkpoint writer walks that same runtime state to build the
* queue-entity list. Before this call existed at all,
* config_activate_committed() was reachable from exactly two
* places: handle_repl_config_push (a STANDBY receiving a push) and
* the boot/recovery path -- never from a member's own live apply of
* the marker it just committed. A live 3-node chaotic-kill soak
* caught the gap empirically: the primary that ORIGINATED a config
* upgrade (do_upgrade_config_with_bytes, which saves the new bytes
* to disk and bumps config_generation but never re-activates them
* into c->myconf on itself) forced a checkpoint at the marker's own
* LSN whose queue-entity section still reflected the OLD
* (pre-upgrade) queue set, then was killed before any LATER,
* independent event (restart, or a racing inbound push) happened to
* call config_activate_committed -- freezing a stale
* queue-entity list into that checkpoint forever (a real
* cross-member materialized-state divergence,
* .wolf/buglog.json bug-1469's byte-level trace: the checkpoint's
* embedded raw config-bytes section correctly carried the new
* bytes, but the SEPARATE queue-entity list section -- built by
* walking runtime state, not the raw bytes -- did not, because that
* runtime state was never refreshed before the write).
*
* MUST run BEFORE persist_checkpoint_marker_triggered(), not merely
* before upgrade_materialization_verify: the checkpoint writer
* (write_snapshot -> collect_queue_frames/collect_host_frames)
* builds its queue/host-entity sections by walking c->qmgr/
* c->workermgr -- LIVE RUNTIME TABLES, populated at boot from
* c->myconf->cluster_config and never otherwise re-synced from a
* later config change on this call path -- not by re-parsing
* mc->cluster_config directly. So activation must settle BEFORE
* persist_checkpoint_marker_triggered() runs, or the checkpoint
* this marker forces still captures whatever runtime state existed
* prior to activation, regardless of how soon afterward activation
* runs. (This ordering constraint is logically necessary given
* write_snapshot's own read path, confirmed by reading
* collect_queue_frames/write_snapshot directly -- not yet
* separately confirmed against a live soak re-run at the time this
* comment was written; see the task's own final live-soak
* verification step for that.)
*
* Symmetric with task 6.2's own fix shape below: one unconditional
* call at the single apply seam every member's marker-apply is
* guaranteed to pass through, rather than patching only the
* originating code path.
*
* Safe to call here for the same reason task 6.2's own call
* (upgrade_materialization_verify, below) is safe at this exact
* seam: neither config_activate_committed nor anything it
* transitively calls (config_committed_marker, local_config_sha256,
* conf_read_current_filename, conf_read_raw, load_primary_config)
* acquires apply_pump_lock, p->apply_pump_lock, or any other lock
* (verified -- config_repl.c's only two mutexes, c->ha.lock and
* c->persist.lock, are both taken in unrelated functions never on
* this call path) -- so no reentrant-lock hazard even though this
* runs while persist_apply_pump_run holds apply_pump_lock across
* its entire dispatch. The disk I/O this performs (WAL segment
* scan, config file read, Cap'n Proto decode) is the same class
* already unconditionally performed at this exact call site by
* upgrade_materialization_verify below (config_committed_marker's
* own scan + a config-bytes read/hash), so this adds no new
* blocking-I/O hazard class. Return value intentionally ignored
* (mirrors upgrade_materialization_verify below): a
* mismatch/absence return (-EAGAIN/-ENOENT) means this member's own
* serve-fence (config_serve_fenced, a SEPARATE mechanism from the
* Activation Fence below) is set instead, and that fence already
* has its own established retry path (config_schedule_activate_retry)
* driven off commit-advance -- this call's only job here is to give
* every marker-apply one unconditional attempt so a member never
* depends solely on a later, independent event to activate.
*
* Guarded on c->myconf != NULL: in production myconf is always
* wired during cluster init, well before any commit/apply can run,
* so this is a no-op check on every real code path -- but
* config_activate_committed's own -ENOENT fallback calls
* load_primary_config(c) unconditionally on c->myconf without its
* own null check (pre-existing in config_repl.c, out of this fix's
* scope to touch), and this call site is reachable from a
* bare/otherwise-empty test cluster_t that exercises
* WAL_CONFIG_MARKER apply mechanics without wiring myconf
* (test_upgrade_snapshot_apply.c's tusa_t fixture,
* test_marker_apply_via_live_pump_checkpoints_at_marker_own_lsn --
* confirmed via a live SEGV in load_primary_config, address 0x28,
* before this guard was added) -- mirrors this same file's existing
* tolerance of a bare fixture elsewhere (e.g. collect_queue_frames'
* `c->qmgr == NULL` check, read_active_config_bytes' "no live
* cluster back-pointer" handling) rather than crashing a test that
* predates and is unrelated to config activation. */
if (c->myconf != NULL)
(void)config_activate_committed(c);
int rc;
#ifdef XCE_TESTING
if (persist_marker_checkpoint_fault_hook != NULL &&