forked from lei.chen/myxce
Rongsong Shen
a5e29a6ffa
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact traffic against a live 3-node cluster, killing the primary mid-stream and asserting zero committed-state divergence and zero phantom fact reactions. Building this scenario surfaced bug-1416: persist_recover()'s boot-time apply-correctness verification can race the live apply pump and the standby live-replication apply path, both of which mutate the same last_applied_lsn/ job-table state the verification re-reads live -- producing a false APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with a new persist_t.recovery_complete gate that defers only the in-memory apply (never the durability write or replication ACK) while a recovery pass is in flight, closed from the earliest safe point in myapp_run() so an election- retry candidate can't apply live traffic before persist_recover() ever runs. Round-2 adversarial review independently verified the ACK/durability paths stay ungated, reproduced the myapp.c fix as load-bearing via revert-and- reproduce, and confirmed real-divergence detection still fires. Live 3-node factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover calls. controller_unit_test: zero new failures across 23 groups. |
||
|---|---|---|
| .. | ||
| config-upgrade-snapshot | ||
| controller-status | ||
| election-iomgr-refactor | ||
| ha-commit-barrier-rollout | ||
| ha-commit-durability | ||
| ha-commit-engine | ||
| ha-config-bootstrap | ||
| ha-consistency-invariants | ||
| ha-convergence-recovery | ||
| ha-elect-ctx-lifecycle | ||
| ha-entity-snapshots | ||
| ha-heartbeat-committed-propagation | ||
| ha-heartbeat-liveness | ||
| ha-peer-connection-reuse | ||
| ha-prevote | ||
| ha-repl-hardening | ||
| ha-single-leader-role | ||
| ha-term-safety | ||
| ha-wal-replication | ||
| job-array-single-wal-record | ||
| libxce-failover | ||
| raft-apply-decoupling | ||
| raft-commit-safety | ||
| raft-drive-async | ||
| raft-ha-core-safety | ||
| raft-library | ||
| raft-log-replication | ||
| raft-sim-conformance | ||
| resval-id-decoupling | ||
| scheduler-controller-msg-gap | ||
| worker-join-refactor | ||
| xce-test-e2e-primitives | ||
| .gitignore | ||