feat(ha-term-safety): out-of-band config bytes, hash-verify-before-activate, serve-fence (task 4.3)

Wires config_committed_marker to a production consumer (config_activate_committed):
verifies pushed bytes' sha256 against the committed marker before activation, fences
(loud, non-serving, stay-alive) on mismatch, and activates the latest committed marker
on recovery. Enforced via ha_dispatch_gate + promotion gate.

Fixes bug-981: a 2nd+ config upgrade could permanently fence a standby on a
push-before-commit ordering. Adds an off-iomgr delaymgr retry scheduled from the
commit-advance hook (atomic read + non-blocking enqueue only on the ACK thread) that
re-activates and clears the stale fence once the new marker commits and bytes verify.
This commit is contained in:
Rongsong Shen 2026-06-22 11:47:15 +00:00
parent c3201a2b1e
commit be4c1f0721
7 changed files with 1074 additions and 12 deletions

View file

@ -119,6 +119,81 @@ int config_append_marker(cluster_t *c, uint64_t version_tag,
int config_committed_marker(const cluster_t *c, uint64_t *version_tag,
uint8_t sha256_out[32]);
/* ── Config byte delivery: hash-verify-before-activate + serve-fence (task 4.3)
*
* The committed WAL_CONFIG_MARKER (config_committed_marker) is the AUTHORITY for
* which configuration is in effect; the configuration BYTES travel out-of-band on
* the existing config-push transport (OPCode_replConfigPush / handle_repl_config_push),
* never inside a WAL payload (R7.4/R7.5). config_activate_committed is the consumer
* that keys that byte delivery to the marker's {version_tag, sha256}: it obtains the
* bytes (already saved locally by the config-push handler when present), VERIFIES
* their sha256 against the committed marker BEFORE activating (R7.8), and on a
* mismatch (or bytes not yet present locally) keeps the member from serving under
* the new configuration and KEEPS RETRYING rather than activating mismatched config
* (R7.9) — the "controller never self-terminates" loud non-serving fence. On
* recovery it activates the configuration referenced by the LATEST committed marker
* (R7.10).
*/
/*
* config_activate_committed: ensure the locally-held config bytes match the latest
* committed WAL_CONFIG_MARKER's content hash, then activate (load) that config.
*
* Behaviour:
* - No committed marker (config_committed_marker == -ENOENT): nothing to
* activate; clears any serve-fence and returns 0 (the legacy bootstrap config
* stays in effect).
* - Locally-held bytes' sha256 == marker sha256: HASH-VERIFIED → activate via
* load_primary_config, clear the serve-fence, return 0 (R7.8/R7.10).
* - Locally-held bytes absent or sha256 != marker sha256: do NOT activate;
* set the loud serve-fence (member stays non-serving under the new config,
* stays ALIVE) and return -EAGAIN so the caller retries when the matching
* bytes arrive out-of-band (R7.9). NEVER activates mismatched bytes; NEVER
* self-terminates.
*
* c — cluster (uses c->myconf for the locally-held bytes, c->ha for the fence)
*
* Returns 0 on activate (or no-op when no marker); -EAGAIN while fenced
* (bytes missing/mismatched); -EINVAL on a NULL/misconfigured cluster; the
* load_primary_config rc if the verified bytes fail to load.
*/
int config_activate_committed(cluster_t *c);
/*
* config_serve_fenced: 1 when the member is serve-fenced under config_activate_-
* committed (locally-held config bytes do not yet hash-match the latest committed
* marker), 0 otherwise. The fence is non-serving + non-self-terminating (R7.9).
*/
int config_serve_fenced(const cluster_t *c);
/*
* config_schedule_activate_retry: when commit advances WHILE this member is
* config serve-fenced, schedule a one-shot OFF-IOMGR delaymgr task that re-runs
* config_activate_committed (task 4.3 r2, Finding D / R7.9). Closes the
* push-before-commit liveness gap: the config MARKER (WAL) and config BYTES
* (out-of-band push) arrive on independent transports, so a standby that saves
* the NEW bytes before the NEW marker commits fences against the OLDER committed
* marker; nothing else re-runs activation post-commit (on_commit_advanced must
* not do disk I/O on the iomgr ACK thread; the primary's config-behind re-push
* net does not fire once conf_save_config advanced the advertised tag). This
* scheduler is NON-BLOCKING (ordq_put only) — safe to call from the commit-advance
* path; the delaymgr worker does the disk reads. No-op when not serve-fenced or
* when the delaymgr is not running (unit-test fixture). Returns 0 on enqueue/no-op,
* negative on enqueue failure.
*/
int config_schedule_activate_retry(cluster_t *c);
/*
* config_activate_retry_fn: the delaymgr task body scheduled by
* config_schedule_activate_retry (runs OFF the iomgr ACK thread, may block on
* disk). Re-runs config_activate_committed; clears the serve-fence once the
* newly-committed marker's bytes verify. Exposed so tests can drive the same
* off-thread activation entry the production scheduler enqueues. `arg` is the
* cluster_t*. Returns the config_activate_committed rc (0 on activate/clear,
* -EAGAIN while still fenced).
*/
int config_activate_retry_fn(void *arg);
/*
* config_commit_change: order + commit a config change through the unified
* (term,lsn) log (ha-term-safety task 4.2 LIVE wiring). Appends a